Thursday, February 21, 2013

Writing for the 98%

How often do users change their default preferences? According to one 2011 study of MS Word users, users rarely do. One could conclude from this that users never change their defaults, and so having user-mutable preferences at all is a waste of effort.

However, this view is reductive! To illustrate, compare the risk of having unintended MS Word settings with having unintended Facebook settings. Both applications will probably continue to function, but one is much more likely to surprise and dismay the user. Thus, users are more highly motivated to change social network settings than editor settings. In fact, according to Pew, 71% of people under 30 self-report changing their social network settings (and 55% of those over 50 do). Perhaps the lesson here isn't that users never change defaults, but rather that users change defaults when it's important enough for them to do so.

It's not always clear what the default for any given preference or feature should be. When the vast majority of users clearly express a preference for one default over another, then the question is easy. When the split is less one-sided, then choosing a default preference can be fraught. This is particularly true when a substantial minority of users are operating under a drastically different risk model. It is hard for one default set of security and privacy preferences to meet the needs of a heterogeneous user base where one person is concerned about invasive network attacks, and one person is concerned about shoulder-surfing.

Which security and privacy preferences are compelling enough for Firefox users to change? Back in December Ilana Segall from Test Pilot team ran a study measuring those preferences that are exposed in about:config and also in the UI. The population for this study is approximately 12000 users who are on the Aurora and Beta channels, and a relatively small number of users who have opted-in by installing Test Pilot on the release channel. In other words, this population is not representative of the Firefox user base, since nearly all Firefox users are on the release channel. In fact, the Test Pilot population is probably much more likely to change settings in general, since they are passionate enough to test Firefox pre-release.


The dates of data collection were from approximately 12/18/2012 for approximately a week, and includes nearly 12000 users. Here's a series of snapshots of the current preferences UI with no changes to default. The preference as named in about:config is listed on the left, along with the percent of users who changed them.


Obviously, no one touches the default security settings very much. There are at least three plausible, possibly-overlapping interpretations: Firefox predicted the most useful default settings correctly, Firefox is doing a poor job converting user actions into saved preferences, or the population who cares about browser security preferences is really that small. The most frequently changed preference on this tab, signon.rememberSignons, controls whether or not Firefox prompts the user to remember passwords. The question of when or why people use the password manager is complex and I'll save it for a later discussion. It is also interesting that fewer people disable Google SafeBrowsing checks for malware than for phishing (browser.safebrowsing.enabled and browser.safebrowsing.malware.enabled). Presumably these are disabled for privacy or performance reasons. Are users who disable one and not the other making a mistake, or do these users consider themselves phish-proof but not drive-by-download-proof? If it is a mistake, why do we allow users to construct a set of preferences that are internally inconsistent in reasoning?

The privacy preferences tab is more complicated than the security preferences tab.
A whopping 11.3% of users enabled Do Not Track (privacy.donottrackheader.enabled). This is an astonishingly high number of users to enable an HTTP header that broadcasts user intent, but is unable to enforce anything client-side. It is a testament to DNT advocates that adoption is this high, but even though this preference is changed by a large minority of users, Firefox should not enable it by default.

A more modest number of users (1.2%) change the autocomplete settings of the location bar (browser.urlbar.autocomplete.enabled) to use a smaller subset of "History and Bookmarks", 1.5% of users have completely disabled browser history (places.history.enabled), and 2.75% of users use custom history settings (privacy.sanitize.sanitizeOnShutdown). The custom history settings are even more complicated:


An astute reader will have noticed that there are already two ways to autostart in private browsing mode: using "Never remember history" and checking "Always use private browsing mode" in the screenshot above which automatically disables browsing, download, search and form history. Approximately 5% of users always use private browsing mode (browser.privatebrowsing.autostart). 0.83% of users have modified their cookie settings to be something other than "Accept all cookies," including 7 hardcore users who don't accept cookies at all. One intriguing explanation is that more users are concerned with local attacks (shoulder-surfing or accidental disclosure on shared devices, by people that they know) than remote attacks.

It is troubling that there are two sets of parallel preferences named privacy.cpd (short for clear private data) and privacy.clearOnShutdown that control custom history settings. Both appear to be used in the same way in the code, but Firefox allows users to enter an internally inconsistent states by maintaining multiple sets of preferences for the same functionality.

Finally let's take a look at the "Encryption" settings under the Advanced preferences tab.
Unsurprisingly, users touch this panel least of all. 0.02% have disabled SSL 3.0 (security.enable_ssl3), 1.6% have disabled TLS 1.0 (security.enable_tls), and 1.0% of users have opted to automatically select a personal certificate (security.default_personal_cert). Is it really worth having a preference panel that benefits fewer than 2% of users overall?

Similarly with the Online Certificate Status Protocol preferences:

1.75% of users have disabled OCSP (security.OCSP.enabled), and 0.03% of users require it (security.OCSP.require). Without knowing anything about OCSP other than its name and a few recent stories about certificate authorities, a reasonable person might conclude that requiring OCSP by default is a good thing. I am guessing that enabling it by default breaks in some cases. However, with so few people requiring OCSP no one is likely to gather enough implementation experience to require OCSP by default for everyone.

For fun I also measured browser.default.searchenginename, the preference that controls which search engine to use when the string in the location bar is not a URL. This preference probably has the most impact on privacy besides the preferences exposed in the privacy panel. Interestingly, it is modified 43.7% of the time (from chrome://browser-region/locale/region.properties, which is how complex preferences get set). The topmost occurring replacements include Ask, Babylon, and AVG, all of which produce interesting search results when combined with "firefox search" on various search providers. One might question if actual user intent underlies changing such an inaccessible preference so frequently, and if not, what to do about it. That, too is a topic that deserves further discussion.

I'll close with a few open questions for the reader:
  • Is there a better way to choose default preferences when a large minority of users expresses a different opinion (e.g., DNT)?
  • Is it worth the engineering effort, UX effort, and screen real estate to make user-visible (to say nothing of discoverable) preferences if fewer than 2% of users benefit?
  • How can we use use this data to make Firefox better?
Thanks for reading!

12 comments:

Anonymous said...

This is great! Did you get numbers on how many users click "Clear History when browser closes"? It could be useful to share that number with the dev.privacy discussion right now.

I am not sure I am convinced by the DNT numbers. TestPilot studies have a bias in terms of technical proficiency. Maybe it makes sense to cross-validate this numbe? a simple check might be just asking AMO what are % of requests have the DNT header set. (Actually, I am not sure, but the AMO blacklist ping should also have this number for ALL users)

I also wonder if changing (back to preferred) the default search is too hard for users(as opposed to switching back the search bar). Did you measure how many times users use the default search instead of the search bar? I wonder if there is a correlation there: users whose default search has been changed switch to using searchbar more often. Nevertheless, something needs to be done.

Monica Chew said...

For "Clear history when browser closes," 0.73% had privacy.clearOnShutdown.history enabled, and 1.86% had privacy.cpd.history enabled. I think both of those should map to the same checkbox, but obviously there is either an error in my interpretation, or the code, or possibly people are manually setting one and not the other. However, it makes sense that these would be lower than "Always use private browsing mode" since presumably users who care about history also care about other session data.

About the DNT numbers, TestPilot users most certainly have a bias. This number is probably much lower in the general population. Alex Fowler showed a DNT dashboard on World Privacy Day based on other data (not AMO blacklist ping, I don't think), I'll have to ask him about it.

About using the location bar instead of the search bar, the most recent data I found is this, but it's incomplete in that we don't know how many people user the location bar for search as opposed to the search bar: https://blog.mozilla.org/ux/2012/06/firefox-heatmap-study-2012-results-are-in/

The 2010 study has people roughly tied for searching on each, with a 3-4% preference for the search bar: https://heatmap.mozillalabs.com/

Thanks,
Monica

Monica Chew said...

Guessing from the decline in search bar usage from 2010 to 2012 (61% to 45.5%) I am guessing that more people use the location bar for search than the search bar. Since Chrome trains users to use the location bar, that would not be a surprise.

Anonymous said...

I think you misunderstood my question.

There are two ways to search: search bar and location bar. As a result, there are two default providers. One is "search provider when you type something in address bar" and the second is "the one in the search bar".

I posit it is easy to switch the default provider in the search bar: there is a drop down list. In contrast, switching the default provider for the location bar is a bit difficult for most users.

What I was wondering was: do users whose default provider for location bar has been switched show a higher tendency to use the search bar?

Based on your reply, it seems your test-pilot study doesn't have the numbers to answer this. It might be interesting to see whether this is the case. This way, one simple way to fix this problem might be to add a "Change Location Bar Search Engine" option in the searchbar drop down.

The Monster said...

Having two different defaults for these is a bad design in the first place. The location bar should either use the same default, or it should look to the current setting for the search drop-down. What is the use case for making them different?

Monica Chew said...

I agree, it seems like a bad design. Also see these bugs.

greggman said...

I think there are more than 3 interpretations. For example how about users that use non-release versions of Firefox only use them for testing and therefore don't set preferences. For that they go back to the release version where they do set preferences.

Nicolas B. said...

We can suppose otherwise :

The users of the beta version use it for previewing and for testing, not for their day-to-day use. So they don't set the software for their personal needs and preferences. Contrary to the users of the stable version.

Monica Chew said...

I find this hard to believe. The number of non-release channel users far exceeds the number of Firefox developers. I seriously doubt that the majority of users in this study were developers who use non-release versions only for testing.

Another point is that Test Pilot only uploads results at the end of a study, after the user consents. It is far less likely that developers who use non-release versions for testing would hit this dialog than people using non-release versions on a regular basis.

Assuming that Test Pilot users are *less* likely to change preferences than release users defies Occam's Razor.

Peter Kasting said...

Hey Monica, has this post been edited? I was convinced that the original version contained e.g. the stat that 0.0085% of users turn on "Master Password", but that stat no longer appears here.

Monica said...

Hey Peter, this stat appears in http://monica-at-mozilla.blogspot.com/2013/02/cant-live-with-them-cant-live-without.html.

Hope you're doing well.

Peter Kasting said...

I just found that the stat is visible in one of the images here even though it's not discussed in the text. Perhaps that's what I was remembering.