In early 2017, a conversation with yet another Waze fanboy finally nudged me to start a navigation app experiment. I was skeptical that the Alphabet owned company could meaningfully best its parent’s home grown Google Maps. I was also curious whether Apple Maps had discovered competence since its iOS 6 release.
I thus set out to answer three questions:
- Which navigation app estimates the shortest travel time?
- How does each app over/underestimate travel times?
- Which navigation app actually gets you to your destination most quickly?
This exercise lasted the majority of 2017 and led me to dread almost any car trip due to the self imposed data gathering tasks that came with it. Nonetheless, my wife and I persevered, and I hope this data serves the community well.
Ideally, I would have tested all routes with two other drivers, each of us departing from an origin simultaneously to follow a different navigation app’s guidance to the destination. This would have resulted in a direct comparison of the apps’ actual performance for all observations. Alas, I lacked two fellow drivers who thought this was a worthwhile endeavor.
Instead, I recorded 120 observations (i.e. trips) of navigation app estimated and observed performance. For each observation, I randomly selected to follow one of the three navigation apps and recorded:
- Which app was randomly selected to follow for each trip
- Estimated driving time for each app (i.e. Apple Maps, Google Maps, and Waze)
- Departure and arrival time
- Traffic conditions (i.e. work commute hours or not)
- Weather conditions (i.e. rain or not)
- Driving type (i.e. >75% city, >75% highway, or mixed)
Sample Data Log
I summarize the results with three simple charts:
Estimated Trip Time: Which navigation app estimates the shortest travel time?
I first looked at estimated trip time. This considers only the upfront estimate from each app and not how long it actually took to drive the proposed route. For purposes of visualizing the data, I indexed performance of Apple Maps and Waze to that of Google Maps.
Relative to Google Maps, Apple Maps estimated trip times were on average 8% longer (i.e worse) and Waze estimates are 3% shorter (i.e. better). These results were largely consistent with what I expected given Waze’s a strong following of users who swear it is the best option among navigation apps. If the estimated trip times consistently predicted actual driving time, Waze would be my preferred navigation app and I believe this is as far as most Waze users get in the navigation app decision process.
Average Error: How does each app over/underestimate travel times?
I next considered average prediction error for each app. For each observation, I calculated the difference between actual observed trip time and estimated trip time, i.e. (Observed Time)/(Estimated Time)-1.
Average error results were the exact opposite of estimated trip time. Using Apple Maps, I on average arrived 1% faster than initially estimated, versus 2% slower with Google Maps and 11% slower with Waze. In other words, Apple sandbags its estimates so that users on average arrive at the predicted time or slightly sooner. Google and Waze are overly optimistic in their predictions and thus their users arrive later than expected.
Error Adjusted Estimated Trip Time: Which navigation app actually gets you to your destination most quickly?
Finally, I combined estimated trip times with estimation errors to derive error adjusted estimated trip time, i.e. estimate for actual time to get to destination. These results are again indexed to Google Maps for purposes of presentation.
Adjusted for prediction errors, not only does Google Maps outperform its competitors, Waze is actually the worst performing of the three.
If you want to get to your destination most quickly, use Google Maps.
If you want an accurate prediction from your navigation app to help you arrive at your destination on time, use Apple Maps.
If thinking you’ll get to your destination quickly helps to ease your commuter anxiety, use Waze.
The performance of the three apps sparks a set of questions regarding incentives and business strategy.
For Apple, Maps is a basic solution for its average user who wants a maps solution out of the box. Apple Maps does not directly drive ad or subscription revenue for Apple so there is less reason for Apple to incentivize iOS users to use Apple Maps over other solutions. However, Apple does care about user experience, and sandbagging trip time estimates so that users arrive at their destination on time results in a great user experience. Hence, I believe that Apple is intentionally conservative with estimated arrival times.
At the other extreme, Waze (Alphabet) makes money through ads when you use their app. What better way to get people to use your navigation app than by over-promising short trip times when no one takes the time to record data and realize that you under-deliver? If an unsuspecting user opens Apple Maps and sees a 34-minute route and compares that to 30-minutes in Waze, the deed is done. Now Waze has a life-long customer who doesn’t realize they’ve been hoodwinked and Waze can throw at them stupidly annoying ads.
“But wait! Waze leads me down super sneaky secret routes that avoid highway traffic jams.” Yea, so… is that a good thing?
Based on various publications, my experience using Waze, and anecdotes from other drivers, I do believe that Waze guides drivers down more “creative” routes. But the results shared above imply that Waze doesn’t get you to your destination any faster. So, is it better to spend 30-minutes following Waze through suburban neighborhoods and alleys than 27-minutes minutes following Google Maps into a highway traffic jam?
I think some people will say “yes — when I’m moving I feel like I’m making progress even if I don’t get to my destination any faster!” And I get that. Driving is stressful and the feeling of progress may help alleviate some commute-related anxiety. But is that progress worth wear and tear on your car and road infrastructure, additional accident-prone miles, and increased traffic in kid-filled suburban neighborhoods that results from Waze’s alternative routing?
Not for me.
The high level limitation of this experiment is the “synthetic” way in which I compare app performance by applying average prediction error to estimated travel times. As described above, an ideal experiment would involve three drivers. Furthermore, the results suffer from additional limitations, e.g.:
- All routes were taken in the San Francisco Bay Area and thus reflect Bay Area traffic, roads, weather, etc.
- 1/3 of the routes were for my specific SF city commute
- Actual route times are heavily dependent on driver (my) behavior, e.g. speed, propensity to pass, use of carpool lanes
- Navigation app routing algorithms constantly evolve, hence this analysis is simply a snapshot in time
Appendix: Data Highlights
- All routes taken in San Francisco Bay Area
- 120 observations — random number generator used to select app used for each route
- Apple Maps: 41
- Google Maps: 41
- Waze: 38
- Driving time:
- Average: 26 minutes
- Min: 8 minutes
- Max: 1 hour, 15 minutes
- Note: to check whether short routes were somehow skewing results I replicated the analysis above with only routes longer than 20-minutes and found consistent results (i.e. Google Maps performs best on error adjusted-basis)
- Observations by driving type:
- >75% City: 68
- Mixed: 20
- >75% Highway: 32
- Observations by traffic conditions:
- Commute / rush hour: 40
- Non-commute: 80