11.11.2020
Mobile

Accessibility in Mobile Apps Through Virtual Assistant, Beyond TalkBack and VoiceOver

Mohit Goenka, Director of Engineering & Nikita Varma, Senior Software Engineer

Yahoo Mail product screenshots

15% of the world’s population has some type of disability. People with disabilities depend on accessible apps and services to communicate, learn, and work. By making apps accessible, you can reach more users. Working on accessibility is like building a skyscraper. After building it completely, it’s hard or nearly impossible to put up an elevator inside if you haven’t planned for it earlier. Teams need to keep accessibility in mind while building features. The 21st Century Communications and Video Accessibility Act (CVAA) makes sure that accessibility laws enacted in the 1980s and 1990s are brought up to date with 21st-century technologies, including new digital, broadband, and mobile innovations. What this means is that people with disabilities can perceive, understand, navigate, interact, and contribute to the web.

In this article, we will share two key ideas that helped to improve accessibility in Yahoo Mail and how these guidelines can be incorporated into any mobile product to make them accessible and usable.

The first is to design apps that work well with screen readers (e.g. TalkBack) and also provide full keyboard support. Next is to integrate the app with a virtual assistant (e.g. Google Assistant), so that users can just command the Yahoo Mail app to perform the desired actions. This not only makes the product even more accessible to people with disabilities but also helps other users who want to get away from a keyboard and want to dictate the system what content should go in an email while composing. Another key benefit is that it improves the performance of certain tasks.

The web is primarily a medium that requires a keyboard and a mouse to navigate and access the product/service, thus making the web inaccessible to many people. Even in the case of mobile apps, touch gestures play a key role in accessing a service or product. Screen reader software like VoiceOver, TalkBack, etc., makes it easier for people to navigate a website or an app if it is developed to work with the screen readers. However, even with screen readers, the user has to gain an understanding of how each screen reader works and learn all the commands to navigate effectively across the application.

As mobile devices are getting bigger by the day and as the content that a user is interested in changes based on an individual’s needs, Yahoo Mail redesigned the mobile apps, both Android and iOS, while keeping user feedback, ease of use, accessibility, and discoverability in mind. The redesigned app works well with screen readers and also provides keyboard support. Apart from the redesign of the app, the team also integrated Yahoo Mail with virtual assistants like Google Assistant and Siri to make it easier for people with disabilities or others to access the product easily with voice commands, thereby making it easier to access the app and the content inside.

Implementation and Redesign

As part of the redesign, we focused on two aspects: visualizing the content of the email that mattered to the user, making it more customizable, easier to discover, navigate, and access, and the ability to navigate the app easily and intuitively with a single hand so that people with limited hand mobility find it easier to access the app. 

Yahoo Mail iOS Inbox

 

The hamburger icon and the sidebar where all system and user folders were present, is now moved to the bottom of the app. This makes it easier to navigate the app as earlier it was difficult to reach the top of the screen with one hand especially in the case of large hand-held devices. 

The bottom navigation bar has Inbox in the first position. When you tap on the chevron icon next to the Inbox folder, it shows a list of all system and user folders. The rest of the views in the bottom navigation bar are customizable. 

For example, the default order for a user in the US would be Inbox followed by Deals, Attachments, Groceries, and more menu which has Starred, Unread, Receipts, Travel, People, and Subscriptions.

In the five-screen attachment above, the first screen shows the Subscriptions view which lists all the subscriptions the user has subscribed to and provides a single tap way to unsubscribe from the subscribers. The next screen shows the Attachments view, where the user can easily find all their attachments like files, photos, and the emails that have attachments in them.The next screenshot shows the Inbox and the bottom navigation bar. The screen following the Inbox is the Deals view where the user can browse and save all the promotional deals they receive in their Inbox. The last screen is the Groceries screen where the users can take advantage of discounts using their loyalty card at any participating grocery store.

Customization within Yahoo Mail

As customization, discoverability, and ease of use were a few of the key areas we focused on while redesigning, the tabs in the bottom navigation bar can be customized. The user can tap on the More menu and tap on customize to drag and drop the views they would like to see in the bottom navigation bar. This helps by letting the user choose the views they interact frequently with and move them to the default bottom navigation bar. For accessibility, the user doesn’t have to navigate several screens to be able to reach the view they wish to interact with. For example, if a user is interested in seeing starred emails, then they can drag the starred view from the more menu and replace it with any other view in the bottom navigation bar.

Theme picker GIF

The number one requested feature from our users was the ability to have a darker theme. Especially for users with low vision, the dark theme provides a complete, soothing, and satisfying experience. When we started redesigning the app, we wanted to make sure we address this concern and as a result, the new Yahoo Mail app provides 14 different color options for themes, all available in light or dark mode. Both the apps provide smart color inversion, that is if a user has iOS13 or Android 10, the dark mode will change automatically with the device.

Some users prefer to have a larger font size for their devices. Both the Android and iOS apps have the ability to support dynamic text resizing and they honor the default system setting for text resizing.

Notification categories GIF

It can be overwhelming when a user gets a notification for each email that comes in. In order to provide personalization and not overwhelm the user, the redesigned app provides users with an ability to choose categories for which they wish to receive notifications for and also the user can select a custom sound. The user can also apply different settings to each of the accounts to help them keep separate. Users can also get active updates on shipping confirmations, real-time package tracking, and flight status updates.

With the redesign, we wanted to ensure that the new design is easily accessible as well. The redesign included several custom views like the theme picker, which has an arc with a knob that can be dragged over to change the theme, and the customizable navigation bar where a user can drag and drop views. It is easier to make text views, buttons, and other simple layout elements accessible. However, as custom views controls require non-standard touch event behavior, additional care must be taken to ensure that they are accessible. In order to maintain compatibility with accessibility services, the code that handles the custom click must generate an appropriate AccessibilityEvent for the interpreted click action and must enable accessibility services to perform the custom click action for users who are not able to use a touch screen.

For people with limited hand mobility, a physical keyboard is the only way in which they can use the app. The most common way to handle keyboard navigation is to make sure the tab and enter key are handled seamlessly. To focus an element using tab, elements must have a tab order specified. For example, in the Yahoo Mail Android app, a user can navigate and use the app by using the tab and enter key. To make sure all elements are clickable using the keyboard keys (tab and enter), the onKeyListener for the given view should be set. Even when keyboard navigation is handled, if there is no visual focus on the active element, it’s nearly impossible for the user to see where the focus is set currently. To fix this issue, we added a visual focus that helps in clearly identifying the currently focused elements. To add visual focus, the onFocusChangeListener should be set for the view. A visual focus in terms of a border can be set on the background resource. Even a 1px/1dp border can make the difference from an invisible element to a clearly visible one.

Integrating with Virtual Assistants

Using screen readers requires knowledge of the tool and all the shortcuts to use the app. Touch gestures, keyboard navigation all require a certain level of dexterity. For a person with little or no dexterity, it becomes challenging even when your app is fully accessible with screen readers. Voice assistants like Google Assistant help overcome that barrier. It’s amazing how these AI-powered virtual assistants can do almost anything a user asks it to. A few examples are emailing, texting a friend, calling someone, scheduling a meeting, setting a reminder, checking the weather, reading the news, giving directions to work, and so on. 

The best part about the virtual assistants is they make the product/app versatile. In fact, to use a virtual assistant, one doesn’t have to invest a lot. If a user has a smartphone, the virtual assistant is usually built-in (Apple Siri) or the user can download a free app (Google Assistant app). It makes interacting and navigating the app easier for all users, not only for those with disabilities. For example, when someone is driving to work and wants to send an important email or respond to one, the virtual assistant can type and send that email for you. 

 

Yahoo Mail Voice Assistant Gif

 

Yahoo Mail has integrated with Google Assistant to make it easier for a user to navigate the app and perform actions through voice commands. In particular, Yahoo Mail has integrated with App Actions, to make everyday basic interactions with a mail client even more accessible and discoverable. App Actions let users launch specific features in your app using Google Assistant. By enabling App Actions to extend your app, users can easily deep link into your apps via Assistant by simply speaking a request to the Assistant.

To integrate App Actions with Yahoo Mail, the team exposed existing deep links to App Actions. When a user invokes an App Action, Google Assistant matches the invocation phrase or query with a registered intent and its corresponding fulfillment. App Actions are like shortcuts, which lets users launch a particular view in the app. They are similar to the Apple Siri shortcuts. Yahoo Mail Android app supports the following core use cases as App Actions:

Create message: Currently, to send an email, one must launch the app, tap on the compose icon and add basic necessary required information (a valid recipient email address) and additional information (message subject and message body).

Search message: Searching for emails is another common use case. Currently, to search for emails or messages from someone or emails containing a keyword, one must launch the app, tap on the search icon, enter a keyword, choose either a contact suggestion from the dropdown shown, or trigger a keyword search.

Launch views: As part of the redesign of the Yahoo Mail app, we added several new features or views that are extracted from the user’s mailbox and shown as views. For example, deals, photos, attachments, travel, groceries, subscriptions are a few such views that users interact with to easily access information that otherwise is buried under 1000s of emails.

All these above-mentioned use-cases are integrated with App Actions and are easily accessible to users with the use of simple invocation phrases and not having to perform any action. The App Actions integration in Yahoo Mail is shipped to all production users and can be accessed using the link: 

https://assistant.google.com/services/a/uid/000019254c5f112c.

To launch these views, one can simply invoke Google Assistant on the mobile device by saying "Hey, Google" or "Okay, Google" followed by the action they wish the assistant to perform. For example, “Hey, Google, send an email to John on Yahoo Mail”,

A few of the invocation phrases supported are as follows:

  1. “Send an email to John on Yahoo Mail with message as hello”,
  2. “Show me messages from Ruth on Yahoo Mail”,
  3. “Search for emails with hello on Yahoo Mail”,
  4. “Open Yahoo Mail photos/attachments/deals/travel/groceries/subscriptions”,
  5. “Find deals on Yahoo Mail”

Impact

Google recently announced that 500 million users are using Google Assistant. In order to enable App Actions, the application needs to support deep links. Enabling the application’s ability to open specific features or fragments could expand them to use built-in intents or custom intents for accessing the functionality directly through Google App Actions. With the implementation of this feature in Yahoo Mail, we are able to serve an even wider range of audiences.

Conclusion

Accessibility in the simplest terms is the ability to access. Accessible design is good design or in other words, a design is good, only when it is accessible. While redesigning the new Yahoo Mail experience for all its users, the team wanted to ensure the redesign is accessible, usable, easily discoverable, and highly customizable based on the user’s needs. Also, the newly designed app has support for various screen readers and can be accessed via keyboard and virtual voice assistants like Google Assistant and Siri. Integrating an app with a virtual assistant increases user engagement and opens opportunities for a broader market. Integrating with App Actions not only makes the features more accessible but also increases the discoverability of the app and increases user growth and engagement with the app.

  • Advertise with us
  • Find the right advertising solutions for your business.
  • Join us
  • Spend every day connecting people with the things they love.
  • Follow us
© 2021 Verizon Media. All Rights Reserved