Value of input not being updated by Dragon NaturallySpeaking

[armoj]

When using Dragon NaturallySpeaking, and the autocomplete dropdown is populated with an answer, the native select form element value is not updated. This prevents the user from continuing to the next page given that answers are required on this particular page.

This is issue was recently found during an accessibility audit we had, and the same issue has been previously documented here by another testing team.

The above article was published in 2018, and the version of accessible-autocomplete that we use is from 2020.

I cannot see anything in the releases after 2020 that relates to this problem. Is this a known issue that is being worked on? I couldn't see anything directly related to this in the issue backlog.

My first approach would be to update to the latest, but as mentioned above, I couldn't see anything relating to this issue in the commits since.

Any guidance on this would be appreciated.

[edwardhorsford]

@armoj is your code deployed anywhere I could try it? can you share a video of the issue?

I could see two likely things that might have gone wrong:

1. The autocomplete isn't detecting the value provided by Dragon
2. The autocomplete is detecting the value, but it doesn't match one of the expected answers

For 2, could you confirm what is populated exactly matches an answer? if not, can you try populating with an exact match via Dragon and confirm it does or doesn't work?

[querkmachine]

Just off the top of my head, this might be the same problem that we found on the character count component where Dragon NaturallySpeaking doesn't trigger any detectable JS events when it enters a value into an input.

It's quite likely that Dragon is updating the value of the input, but Accessible Autocomplete doesn't have any way to know that the value of the input has changed, so the underlying select is never updated.

[armoj]

Just off the top of my head, this might be the same problem that we found on the character count component where Dragon NaturallySpeaking doesn't trigger any detectable JS events when it enters a value into an input.

It's quite likely that Dragon is updating the value of the input, but Accessible Autocomplete doesn't have any way to know that the value of the input has changed, so the underlying select is never updated.

That is the behavior that was evident during the testing. It updates the autocomplete, but the form element is left unchanged, evidenced by the page throwing an error due to the field being blank upon form submission.

[edwardhorsford]

@querkmachine we built the Accessible Autocomplete specifically to handle this - we had a fix that worked. I believe the solution was to regularly poll the value. It's possible that Dragon has changed something to break this, but it could be a number of other issues.

[edwardhorsford]

Just off the top of my head, this might be the same problem that we found on the character count component where Dragon NaturallySpeaking doesn't trigger any detectable JS events when it enters a value into an input.
It's quite likely that Dragon is updating the value of the input, but Accessible Autocomplete doesn't have any way to know that the value of the input has changed, so the underlying select is never updated.

That is the behavior that was evident during the testing. It updates the autocomplete, but the form element is left unchanged, evidenced by the page throwing an error due to the field being blank upon form submission.

@armoj there's a number of things that can cause the select not to get updated - such as the value not exactly matching as I alluded to above. Can you share details of your implementation?

As a solution to get you going, you can also update your configuration to additionally send the value of the autocomplete input (not the hidden select) to the server. Eg you don't need the select to get updated. Your server can then interpret this as valid or not and proceed. This will mean you can also get invalid input / typos and would need to handle these with validation.

[armoj]

This testing was done by an external team, so I am unable to replicate the issue myself. I could have a look at your suggestion, but I am sure you would understand that it is not the most ideal fix.

What I don't get is what is meant be "exact match" in this situation. If the voice recognition recognises a word that is on the list, then selects that value in the autocomplete, is that not deemed an exact match? If it is, why would the underlying select not be updated?

Conversely, if it is not an exact match, why is it matching something in this case?

Is there a situation where there can be a loose match that is "similar" to one of our predefined values? And in this scenario, it doesn't update the underlying select element?

[armoj]

@armoj is your code deployed anywhere I could try it? can you share a video of the issue?

This is our production service:
https://claim-criminal-injuries-compensation.service.justice.gov.uk/apply

The page with the autocomplete on it is a page where you choose a police force. To get to that page you need to go through a handful of initial questions. Please make sure that you answer using the following:

• Apply for myself
• Applying because you are the victim of violent crime
• Has been reported to the police
• you are over 18

Please make sure you don't go to the end of the application and submit it please! 👍

this should get you to the police page. Please let me know if this works and you can use Dragon on it.

[armoj]

The external team have provided me with steps to reproduce the issue:

• Say “ tab” till focus moves to the “change location” link
• Say “Press enter”
• Say a location e.g. “Bedford” and then “Press enter”, then the page will refresh with the location selected.
• Alternatively, say “press down arrow” until the desired location is highlighted then say “press enter” and the page dynamically updates to have the new location

[querkmachine]

@armoj Am I understanding correctly that you have it set up so that a page navigation/refresh takes place as soon as a value is selected from the autocomplete?

If that's happening then there's a decent possibility that the polling that @edwardhorsford mentioned doesn't have time to run before the user leaves the page.

[armoj]

@armoj Am I understanding correctly that you have it set up so that a page navigation/refresh takes place as soon as a value is selected from the autocomplete?

No that is not the case for this (or any of our pages). No JS-driven refresh is carried out when things are selected. Just performs as a standard form with a continue/submit button

[querkmachine]

@armoj Alright, thanks for clarifying 👍

[edwardhorsford]

From memory, the polling is 10 or 100 times per second. My gut would be that speed isn't the cause here - but easy enough to test to rule it out.

@armoj I find the steps to reproduce the issue confusing - you say there's no js-driven refresh, but the the steps somewhat imply that's what's happening. Could they be expanded? The steps to reproduce also imply things are working correctly - could you include what was expected vs what happened?

[edwardhorsford]

This is our production service: https://claim-criminal-injuries-compensation.service.justice.gov.uk/apply

From looking at the deployed thing, something seems slightly broken to me. When I partially type an answer, the first option in the list is selected as if 'auto-select' is enabled, including showing the selected item in grey as a hint:

If I then if I tab out of the field, that answer is not populated. With auto-select enabled, I'd expect the selected thing to be populated when exiting the field.

With that said, for this use case I would recommend not enabling auto-select as this may incorrectly route users to the wrong police force. You're probably better forcing them to make a choice, or else showing them either a helpful error message or results 'close' to what they typed.

I think this behaviour points to a bug somewhere. Is your repo public somewhere?

[edwardhorsford]

What I don't get is what is meant be "exact match" in this situation. If the voice recognition recognises a word that is on the list, then selects that value in the autocomplete, is that not deemed an exact match? If it is, why would the underlying select not be updated?

A rambling answer below that doesn't necessarily help solve this problem, but might explain some of how the autocomplete works.

There's a few different ways a user might interact with the autocomplete. A user typing might type a few characters and then pick from a list (as you describe). But a voice user is more likely to dictate an entire answer in one go. Rather than L, O, N, D... they'd dictate "London". This means they're less likely to get some of the benefits of autocompletes as they've already given you an answer. You'd need to make sure you have sufficient synonyms set up so that if they give you something slightly different, the autocomplete can still provide helpful suggestions.

For example, if I type "London" there's only one result. I'm sure there's more than one london force though! (have since found them under 'metropolitan').

As I don't think you've set up synonyms, a lot depends on whether what they've dictated matches against one of the strings in your list. eg does "London" exist in your list of results? The default matching algorithm in the autocomplete isn't great, so requires the exact typed string to exist in your results. So a search for "City London" will fail as that doesn't exist - only "City of London Police"

Auto-select also plays a role here. Continuing with 'city london' as an example. If a user dictates that, currently there are no results. With auto-select enabled, when they blur the field I'd expect their answer to get wiped because it didn't match any results. With auto-select not enabled, I'd expect the field to remain populated with 'city london' - though note this doesn't match anything, so the underlying select would either be empty or pointing to the incorrect previous answer.

If they'd dictated 'city of london': there's one result. With auto-select, blurring the field will select it - and should do too in the underlying select. Without autoselect, blurring the field will leave it with 'city of london' which looks correct, but doesn't match an option and leaves the underlying select either blank or with the incorrect previous answer. What they'd need to do is actively click on the 'city of london police' option, or else make sure it is selected and press enter.

Conversely, if it is not an exact match, why is it matching something in this case?

I'm not entirely sure what you're asking here.

Is there a situation where there can be a loose match that is "similar" to one of our predefined values? And in this scenario, it doesn't update the underlying select element?

You can set up the autocomplete with synonyms / other things to match. This is a good thing to do as people search with all different sorts of terms. If a force is known by multiple names or perhaps abbreviations, it's good to add synonyms to help people find the right one. Eg you might want to add more variations so a user typing 'london' gets a list of all the forces in London.

In general using the autocomplete has these stages:

• User provides a search query
• Based on this query, the autocomplete can display 0, 1, or multiple matching results
• If auto-select is enabled:
• • Pressing enter or tab will select the first result, regardless of what it is. If there were 0 results, what's written in the input (their query) is cleared. If there was a result, it'll be changed to the name of the result. So if they'd typed 'england' and that matched to 'United Kingdom', that's what will be shown in the input.
• • The underlying select should get updated (except possibly where there were no results, where I have a hunch it might stay on the prior value -but haven't tested this)
• If auto-select is not enabled:
• • Pressing enter or tab without picking an item will leave the input with their query. The select will remain in whatever it's previous value was (which may be blank, or may be a prior answer!)
• • If they pick an item, the input will update, as should the select.
• • If their query exactly matched an answer they might just tab on. I don't think the underlying select gets updated in this case.

Aside: It's sort of a bug / undocumented thing - but you really need to submit the value of the autocomplete input to the server as well as the select and compare the two. There are situations (described above) where the user can have picked an item, then changed their mind and typed something else, but the autocomplete won't have updated the original select. You either need to solely use the value of the autocomplete input and process it server side, or else compare it to the select and make sure it matches.

It's generally not safe to rely solely on the value of the select when the js component is running. The docs for the autocomplete don't really mention this, but I say this from much experience working with this component.

[edwardhorsford]

The external team have provided me with steps to reproduce the issue:

• Say “ tab” till focus moves to the “change location” link
• Say “Press enter”
• Say a location e.g. “Bedford” and then “Press enter”, then the page will refresh with the location selected.
• Alternatively, say “press down arrow” until the desired location is highlighted then say “press enter” and the page dynamically updates to have the new location

I think two different interactions are described here, but it's not clear if both are broken or just one, nor what the outcome was vs what you expected.

---

When they say "Bedford", any results are shown by the autocomplete? If results are shown, I think it implies the autocomplete is successfully polling for the typed query. When I do this by typing, I get one result. Pressing enter selects that result and changes my query from "Bedford" to "Bedfordshire Police". Do they get results? does pressing enter transform the query to "Bedfordshire Police"?

Note that normally a user would need to explicitly navigate with arrow keys or else pick the correct option, then press 'enter'. That it's pre-selected implies auto-select is enabled - but note my prior message this appears to be partially broken, so I don't trust what's happened here.

[edwardhorsford]

Aside: once you've got this bug worked out, you might like to look at this blog post I wrote last year about how you can make the autocomplete significantly more usable by swapping for a different matching algorithm. Code is linked in the post.

[armoj]

Thank you for your comprehensive replies! I will take my time and go through them.

In the meantime, our repo is public and here is our autocomplete implementation:

• Autocomplete wrapper module: https://github.com/CriminalInjuriesCompensationAuthority/cica-web/blob/master/src/modules/autocomplete/autocomplete.js
• Autocomplete wrapper initialisation: https://github.com/CriminalInjuriesCompensationAuthority/cica-web/blob/master/src/js/scripts.js#L22
• Autocomplete module directory: https://github.com/CriminalInjuriesCompensationAuthority/cica-web/tree/master/src/modules/autocomplete

From my understanding of the issue that Dragon was having, this problem could be solved by introducing a comprehensive list of synonyms for relevant entries. autocomplete is set to trye in our implementation as seen here.

So, for example...

if "bedford" was a synonym for "bedfordshire police". That way, if "bedford" was entered, then the user tabs away, the first match, which would be "bedfordshire police" would be selected and populated in to the native select element due to the synonym.

And to clarify, in this specific scenario, the native select would be populated with the expected value of "bedfordshire police", if either they tabbed away or clicked the result. Is this correct?

[edwardhorsford]

Without more details about what's going wrong, it's hard to say what the problem is. Ideally a video and /or fuller details of what's happening are needed.

I don't think lack of synonyms are at fault, but without knowing what was tested, it's hard to tell. From the details you've given so far, it's unclear if the autocomplete is picking up what was typed and displaying results or not.

---

Synonyms are good to have, and will greatly improve the usability, but probably not the cause of your issues. For the specific case of "bedford" - if a user types 'bedford' they'll already get suggested 'bedfordshire police' as 'bedfordshire police' includes the string 'bedford'. However if they typed 'bedford police' they won't get any results.

There are probably other cases where you'd benefit from synonyms - like the london boroughs, which I initially couldn't find.

And to clarify, in this specific scenario, the native select would be populated with the expected value of "bedfordshire police", if either they tabbed away or clicked the result. Is this correct?

This would depend on whether auto-select is enabled. Your code has it enabled, but it doesn't seem to be working as expected. Looking at your code, I think it might be because you have cofirmOnBlur disabled. It's been years since I looked at that option, but have a feeling that you want to have it on when using auto-select (the docs suggest it defaults to on).

As noted above, you cannot rely on the value of the select in any case - for a robust solution you'll need to send the value of the autocomplete input to the server and compare the two answers.

---

If you do consider adding synonyms, I'd highly encourage you to look at the blog post I linked above and the attached code. We wrote the code as a 'drop-in' replacement. Rather than the in-line source function you're using, you call our code instead - you'll get a bunch of extra features that improve usability and make doing things like synonym matching easier.