The first time I witnessed the challenges of disability was when I started college in 2016. My hostel mates came from across Asia and Africa, pursuing different career paths and hobbies, each with their own cultural identity and lived experiences.
Back then, an entire wing of my hostel was dedicated to blind students and peers with limited vision. Running into them in the alleys, en route to the dining halls, or toward the college — and guiding them to their destination, hand in hand — quickly became a daily routine.
The short conversations were always fascinating, and their memorization of entire volumes of Urdu poetry never ceased to amaze me. During the exam season, I regularly volunteered as a writer while my friends narrated the answers.
It was a revelatory experience for me, yet two aspects stood out. First was the permanent presence of smartphones in their lives. Second, despite being a deemed national university, there was an utter lack of disability assistance, especially for blind residents or guests on the entire campus.
These shortcomings range from unplanned infrastructure to a lack of a supportive tech stack. Saif Khan, an architect, tells me that there aren’t any standard directives in place to make buildings accessible for blind people. “The best we do is build ramps for people with motor disabilities,” Khan, who owns Pause Design Studio in India’s national capital, tells Digital Trends.
The situation is grim for multiple reasons. Dr. Arif Waqar, who has worked extensively with blind people, tells me that even in medical science academia, the focus is more on the curative side rather than solving the existing problems.
“We are not banking on the technical innovation side. And that means real-world problems continue to persist without a universal solution. Navigation assistance is one of them,” Waqar says.
A new way to assist blind people
A team from the University of California, Santa Cruz, wants to use smartphone apps to help blind people navigate buildings. Notably, these applications do not require any pre-existing technical setup and need only a phone’s internal sensors. Not even the camera module is part of the equation here.
Roberto Manduchi, professor of computer science and engineering at UC Santa Cruz, spearheaded the development of these apps to help blind users move within a building using audio cues. The most convenient — and safest — part is that users don’t need to hold the phone while at it.
Think of these two apps — named Wayfinding and Backtracking — as the GPS equivalent for indoor navigation. However, unlike other attempts that require pre-installed sensors in buildings or non-reliable GPS, these apps need only the sensors fitted inside a phone to offer guidance.
These apps need only the sensors fitted inside a phone to offer guidance.
Specifically, the team used readings from the inertial sensors – the accelerometer, gyroscope, and magnetometer — to measure the navigation progress. Though mainly reliant on smartphone speakers for vocal cues, the apps can also be paired with a smartwatch.
Five meters ahead of each turn, the apps instruct the user about the upcoming change in direction. While the Wayfinding app helps with entry and navigation, the Backtracking app uses the blueprint of the initial journey and simply reverses it to provide the necessary guidance.
In the near future, the team hopes to integrate computer vision tech into the apps. That would allow users to click a picture of their surroundings when they find themselves in a tricky spot and have the AI describe the world around them.
The vision is similar to how modern AI chatbots can now process images, allowing users to simply point the camera and let the AI make sense of it.
How it all works
As part of the tests, the team at UC Santa Cruz had seven blind participants who used the Wayfinding app to move past routes, which had 13 turns in total. Next, they used the Backtracking app to retrace their original path on the return journey.
These apps are not reliant on any external infrastructure, nor do they require users to hold the phones in a certain position to capture data from their surroundings. The apps work fine even when the phone is tucked safely in the pockets.
This is a crucial victory from a convenience standpoint. “Blind travelers normally use a long cane or a dog guide, and thus have one hand already occupied maneuvering the cane or holding the dog,” says the research paper, which has been published in the ACM Transactions on Accessible Computing journal.
This is a crucial victory from a convenience standpoint.
For the Wayfinding app, the team tested two separate algorithms: Azimuth/Steps and RoNIN. The first one borrows from the concept of step tracking, creating a two-dimensional step vector at each recorded step and directional information pulled from the phone’s onboard compass.
To estimate the user’s position, a “dead reckoning” system is applied. “This is akin to the old navigation technique that computed a ship’s path using the compass for heading and a chip log (a rope with a number of regularly placed knots) for speed,” Manduchi explains. “You reconstruct the ship’s path by drawing a line on the map based on measured heading and speed.”
Algorithmic errors — or “drift” — are inevitable. To counter that, the team used a technique called particle filtering, which is mostly used for spatial tracking. In this case, particle filtering was used to add certain corrective constraints to avoid this drift.
“We cannot cross walls (unless we are Superman). Adding these prior constraints (drawing from the underlying floor plans) dramatically reduces the effect of drift,” Manduchi tells Digital Trends.
The RoNIN algorithm was mostly deployed as a failsafe and comparative analysis, and during the course of testing, it was only required once. For estimation of the shortest route, the apps rely on Apple’s GameplayKit system, a framework predominantly used for building games. Notably, Apple already has an indoor map framework available for developers.
“We cannot cross walls (unless we are Superman).”
The team also used smartwatch-based controls on an Apple Watch, using a mix of controls including touch-based swipe, Digital Crown movement, and VoiceOver. The apps alert users about the next turn, wrong movement patterns, nearby landmarks, and when they enter a new route segment.
The team expressed confidence in the inertial-based localization technique driving the app, predominantly because it’s accessible and doesn’t require any external infrastructure to offer guidance services. However, Manduchi tells me that Wayfinding and Backtracking are “only experimental apps, still far from a distribution version.”
Notably, the team is considering the open-source route via the UCSC Center for Research on Open Source Software platform. However, a public release might take some time in order to address “several practical issues,” Manduchi says. Unfortunately, one of those issues is fundamental in nature.
The big caveat
Convenience is a recurring theme behind the two apps, and it seems there are no demanding hardware limitations. I asked if the apps required a certain level of silicon firepower during testing or if the finished app had a performance baseline.
Manduchi tells me there are no such performance expectations from the native hardware, as the team tested the two apps on an iPhone X, which came out eight years ago. It is safe to assume that when the two apps are released publicly, most iPhone owners will be able to run them with ease.
Now, the Wayfinding app is driven by the availability of floor plans. It would be useless if the app didn’t have the floor plan already saved in its directory. More specifically, the building floor plans need to be fed in a vectorized format.
Manduchi’s team has already solved one side of that crucial hurdle. “We created a web app that facilitates vectorization of a floor plan in any existing format, such that it can be used by our app,” Manduchi tells Digital Trends. He says they are looking to release the web app publicly in the near future.
The real obstacle is the availability of these building maps, vectorized or otherwise. “Our experience is that availability of floor plans of public buildings is spotty, and available floor plans can be in various formats,” Manduchi tells me.
That problem can only be solved with voluntary participation or if the state decides to intervene. “As an architect, I give my clients the floor plan or digital sketch map in PDF format, because that’s what they need to see. Realistically speaking, the hefty vector maps in IMDF files are of no use to them,” Khan tells me.
He emphasizes that there is no such thing as a hardline trade secret involved here, but most architecture firms or entities won’t share the 3D vector maps. “Think of it like an intellectual property, something that is the foundation of the work I did,” Khan tells me.
These vector maps are usually created in demanding apps like Revit or AutoCAD and can only be viewed or manipulated using those apps. So, it makes little sense to share them either way, as the client only needs a floor map that they can open and comprehend, like drawings on a simple PDF or digital sheet.
Even if the current owner of a certain building agrees to supply a copy of the map for apps like Wayfinding, they would first have to track down the architect or planning firm behind it. They may or may not be able to share for logistics, contractual, or any other legally protected reason.
“These apps at least offer a viable route fix to walk past those mistakes.”
However, the situation becomes even trickier for buildings run by government entities. These could be anything from the nearest public hospital to the local metro train station. Applying for a permit and getting it approved to access the vectorized maps could be a long-drawn-out and tedious process.
In India, for example, university campuses, public infrastructure, and government offices were built during the British colonial era. Finding a map or even an architectural floor plan would be like finding a needle in a historical haystack.
The only path forward would be to digitally remap them professionally, which would be another massive undertaking. For now, open-sourcing seems to be the only meaningful path ahead, as that would at least ensure that Wayfinding and Backtracking apps can provide navigation assistance in any meaningful capacity.
“There is no architectural provision for blind people. There are no enforceable guidelines,” Khan tells me. “These apps at least offer a viable route fix to walk past those mistakes.”
