Mobility metrics and indicators rethought
Author: Gérald Gurtner
Performance is about comparing some output of a system with some level of expectations. The issue of setting the right level of expectations is certainly a major issue by itself, but choosing the right metrics to measure is probably even more difficult.
This difficulty comes from the fact that Key Performance Areas (KPAs) live in a different world than Key Performance Indicators (KPIs). KPAs live in a qualitative world, where general ideas are thought to be important for human beings. For instance, ‘safety’. KPIs on the other hand belong to a quantitative world of ‘cold values’ — floats, integers — observed on the real world. Matching these two worlds is like getting into Mordor: first you think that it will be obvious, then you think that it will be impossible, and you finally pick a way because it is pretty much the only one available.
Indeed, the potential KPIs that one could imagine are fortunately severely restrained by reality and what we can observe in the system. For instance, in DATASET2050 we were trying to define an indicator for the ‘seamlessness’ of a trip, something which is important for all travelers without a doubt. Important, ok, but what is it exactly?
Seamlessness is about the perception of travellers. As a consequence, it is highly subjective, which by definition cannot be part of an indicator, because an indicator is meant to be objective. So instead of a top-down approach where we use the question ‘What would be the best metrics to measure in order to represent seamlessness?’, we are left with a bottom-up approach consisting in ‘Among the ones I can measure, what are the metrics which would be related somehow to seamlessness?’.
So, what can we measure? For many years now, sociologists and psychologists use the ‘cognitive load’ to have a measure of the effort needed by a brain to accomplish a given task. Seamlessness is about being able to forget the trip itself and not actively be forced to take decisions or looking for information for the continuation of the journey. We thus defined a first indicator, which is the total cognitive load of a given trip for the passenger as a measure of seamlessness. Ok, but how do you measure cognitive load in reality?
Well, you don’t, as least not on a large scale. And here comes the second step of the search for a good indicator: can we find something easily measurable which is an approximation for what would be a perfect indicator?
In the case of seamlessness, we have to go back to how the travel unfolds. For instance, what is the difference between:
1) depart from home, take a taxi, take a train, take a taxi, arrive at destination.vs:
2) depart from home, take a taxi, take a train, take another train, take a taxi, arrive at destination.
Easy: there is one train more. Ok, but what makes you choose the first option over the second if both have the same travel time, price, etc.? Well, the first is easier, right? You do not have to think about getting off the train, find the next one, wait, get in train, possibly struggling to find a spot to seat, etc. So the idea that the first one is easier than the second one comes ultimately from the ‘continuation’ property of the actions you are taking, which is associated with a low cognitive load dedicated to the journey. In other words, taking different actions during a trip is more annoying that taking only one action.
Following this idea, DATASET2050 defined the journey as a series of ‘phases’ and ‘transitions’. ‘Phases’ are typically long with a low cognitive load dedicated to the journey, whereas ‘transitions’ are short and require the active participation of the passenger in order to continue the journey. A simple indicator can then be defined as the number of transitions taken in a single journey, which is trivial to compute for nearly any journey, with very little data input.
A slightly more advanced indicator is to consider the time spent within the transitions — for instance, queuing times — compared to the total travel time. For instance, a small 45 minutes trip where one has to take three buses is quite tiring compared to a single-bus journey. This indicator requires more data, as the specific times in each of the segments are required. However, it is largely feasible to compute it with modern methods of data collection (e.g. GPS tracking). Giving a good balance between the measuralibity and its concetpual proximity with the initial KPA, this indicator is the one which has been selected as key performance indicator for seamlessness in DATASET2050.
In DATASET2050, we have gone through the exercise of finding the right indicator for all of the KPAs defined by ICAO, including safety, flexilibity, efficiency, etc. These concepts are sometimes too vast and need to be broken down into sub-KPAs, called “Mobility Focus Areas”. For all of them, several indicators have been defined, but we selected only one final KPI in the end per KPA. For instance, the KPA “flexibility” has been subdivided into “diversity of destinations”, “multimodality”, and “resilience”. Only on key indicator has been selected in the end, weighting the travel options by the distance between the potential destinations. All this work can be found in the public deliverable 5.1 of DATASET2050, soon available here
To conclude, the choice of a good indicator is thus dictated by the balance between the measurability of the metrics and its relationship with the overall concept. This is an important issue, as the indicators are then used by the policy makers to drive the system is a certain direction. And the quality of the indicator decides whether it is the right one or not.