Lots of hype today over a new tool someone released that digs into a database iOS 4 devices create. I fired it up this morning, after decrypting my iPhone backup, and it did pretty much show the areas I've been in since iOS 4 came out. This included a nice streak across the US when I drove from Austin to Southern California late last year.

I've been looking into it deeper today, and figured some of the info might be of interest to others here. First up is a response Apple sent to the House of Representatives last summer. Page 6 talks about the data collection, basically Apple switched from using Google for cell triangulation and Skyhook for WiFi to their own database in iOS 3.2 and above.

People looking at the sqllite database directly have noticed the coordinates seem to be the locations of WiFi base stations or cell phone towers and not the location of the phone at the time. Some attendees of WWDC 2010 have commented that Apple talked a bit about this file, and the purpose is to cache location information for power savings reasons. The idea is that the phone has to do far fewer calls over the network to get this data, especially for people who use location services in the same areas frequently.

Not really sure what I think about this yet, beyond curiosity to see what the data reveals if I map it out with more precision. An update to iOS may start wiping this data from time to time depending on how severe the outcry is.

It's not really a privacy concern any more than the information you enter into your own phone. The issue here is that there isn't a setting to control the behavior or to manually wipe the data. I suspect/hope we might see something like that in the future. Something else Apple might do is eliminate any association of the data with dates.

The data on mine definitely shows very rough areas I've been "around" but it's very far from exhaustive. It also displays a start date earlier than my iPhone purchase. The points on the map are a lot more localized than the areas I've actually been to with the phone.

Couple it with a Cellebrite-equipped cop pulling you over and you might be more concerned about privacy.

http://www.myfoxdetroit.com/dpp/news/sou...k-20110420-wpms

If I get arrested in Michigan, my first call will be to someone who has my iTunes password so they can remotely wipe my phone. Thanks for the heads up.

From what I've been able to dig into, turning off Location Services entirely would disable this, but I'm not sure if it clears the existing file or not. May have to play around with that and see what happens. The downside of course is that all location based services do get shut down with that switch, so not even the basic Google Maps would function.

My current iOS devices are set to wipe the device if the unlock passcode is not entered properly after a certain number of attempts. This is mostly to protect work data that is on the devices. Unfortunately I'd likely not have time to blindly trigger this, as the phone goes into a lockdown for a minute after 5 failed attempts, and only a few more failed after that would initiate the wipe. It is really a shame that data privacy in the US is so weak these days. Even though I remained within the US, the Border Patrol had the right to search my laptop when they stopped me during my drive from Austin to SoCal. Do I have anything incriminating on the laptop? Not that I'm aware of, but I also don't want other people just randomly searching it. Nor would I want a cop in Michigan scanning my smartphone with a Cellebrite.

So what stops "them" from dissembling it and going for a lower level readout?

Much clearer post from someone who has already done a field trial to see what was tracked.

http://www.willclarke.net/?p=247

Definitely only recording where cell towers and wifi spots are, and not the exact location of the phone. Still potentially bad that it shows what areas a person has visited, but only if the phone is taken and plugged into a computer or Cellebrite type device.

I'm not sure how feasible it is to dismantle an iPhone or iPad and read data directly off the flash chip. My current understanding is that with the 3GS and above, all data on that flash chip is always encrypted, but I don't know how secure the key is.

It's currently easier with the iPhone to just jailbreak the device to get around the protection, even with the latest 4.3.2 release. For now, the iPad 2 has yet to be broken in that way, so I put a little more trust in it.

Security wise with my company, a compromised phone would at most get someone access to my e-mail. Anything worth accessing is protected in other ways that can be locked down quickly. E-mail could also be wiped, leaving someone with just the cached past few days worth of messages on the device. I do also have both active with the Find my iPhone service, allowing a wipe command to be sent over the air.

With recent devices I believe the first thing iOS does is delete the key that the rest of the data on the device is encrypted with.

...assuming that your phone can be reached over the air, no? Surely all they have to do is put your phone in a Faraday cage or similar?

Or put it in Airplane mode, turn off the 3G radio and make sure it's not on their WiFi network, turn off the FInd my iPhone feature (if it's not password protected), or turn off the phone.

The market for faraday cage resale isn't as big as one would think.

I don't have a particular paranoia about cops or other government agents searching my things, it's mostly an annoyance about the processes they are allowed to do (such as potentially scanning the entire contents of a smartphone during a routine traffic stop). If I were ever in a situation where they were deeply searching my things to the point of dismantling them or defeating the first layer of security, I probably have bigger issues to worry about at that point.

Most of my worries are more about what happens if i lose a device, or if one is stolen. There are plenty of ways to stop the wipe command from arriving, but I'm not as concerned about the device if they have cut off all network access. It's mostly a concern about what anyone could access directly using the device, requiring it to be online and also receptive to the wipe command. Today, that concern is mostly related to any work related things on the device. Down the road with payment options and other uses of the phone, it will mostly be a desire to prevent fraudulent charges. I already use my phone today to pay for things at Starbucks. I could easily see it replacing my full credit card in the next few years.

Senator Al Franken* wants to know what the deal is.

I don't think this sort of thing surprises any of us when it's discovered, and that makes me kind of sad. I expect phones to have these sorts of features, but it's very "Apple" of Apple to bake this tracking "feature" in with no prominent mention of it or explanation of its purpose and no way to turn it off.

I'm interested to hear back about what they're doing with the info.

* I still get a kick out of that.

I think it's best described as a "caching feature" rather than "tracking feature." It doesn't actually track you, not only because it doesn't mark your position, but also because none of that information is transmitted anywhere.

Also, it was apparently openly discussed last fall. This week it made a bigger splash in the media because of the app that visualizes the data.

If it was only a "caching feature", there would be little need to retain all of the data indefinitely, as is apparently done.

It could very well simply be something that wasn't implemented - a purge feature. I'll bet money right now that it's coming though. Obviously because the issue has been brought to the mainstream.

None of this is an excuse for Apple not having been more forthright about this, nor for not including a method to clear/purge the data automatically/manually or turning the feature off entirely.

I believe the consensus at this time is that the data helps the device to save battery. Somehow.

My data is nowhere near as dense as some of the examples I've seen others post. No Oakville, barely any Mississauga, no Toronto, no Montreal, no Tremblant and nothing South of Burlington - all places I've spent significant amount of time connected to 3G and passing by wifi hotspots. And honestly, I've not spent any significant amount of time at the 407 and 400 where you see that cluster in Vaughan. At most, I can remember driving through along the highway in fact.

Yeah, I linked to an Ars story about Franken, just didn't spell it out, and I do share some amusement in him being a senator now. Will be interesting to see the response this time, since the one provided to Markey last summer was a handy quick summary into what Apple was doing location wise, and much easier to parse then the full EULA.

Someone posted this on reddit


The EULA does point out on page one how to disable anything location wise, and I'd assume it includes the population of the local cache. Will have to test myself if it is wiped if the switch is turned off, or just frozen at that point. Noone seems to be clear on that.


The current iOS 3.2 implementation (it shipped on the iPad 1 first) was known about for a while, and the previous implementation has also been known about. More on it can be found here, written by Alex Levinson, one of the people who first started investigating it.

And the same thing has been found on Android phones, with one difference in how the data is retained. Android will only cache 50 cell phone sites and 200 wifi spots, and when it hits the limit, the oldest entries are removed to make room for the newest.

https://github.com/packetlss/android-locdump

Interestingly, the source code that manages this moved from being part of the open side of Android over to the closed side at some point.

Thinking about this more, an easy solution for the tracking fears would be to replace this cache the phone creates for AGPS with pieces of the database directly from Apple when syncing/updating the phone. It could do a quick location update, and seed the database with information in a 100-200 mile radius. Not sure how large the DB would be for that, but the size could be adjusted to still maintain quick performance. Maybe even let the user pick a general area, for frequent travelers.

Not sure if Apple would be willing to do this though, since the master lists for WiFi access points are quite valuable. Skyhook's sole purpose for existing is to generate these lists, and sell access to them.

Amusingly people will still happily tweet and post to facebook.

Yes, it's a bit daft of Apple to allow that level of caching, but real world impact... not so much.

Seems the echo chamber continues to hype this one, to the point where two users have decided to file a lawsuit.

The Wall Street Journal also is reporting that turning off Location Services still causes the iPhone to collect the location info for cell towers and WiFi spots. I however found the opposite in my testing with a freshly wiped 4.3.2 iPhone. In my test, I did have an initial cache created, but it never added any data even when I opened Maps with location off. If I turned location back on, the cache updated. I did the test by driving the same route south of my house twice, the first time with location disabled. I did a sync to a laptop not normally used, wiping the backups each time to force iTunes to do a full backup each sync. I chose to head south due to the initial cell data in the database showing cell towers up to 25 miles north of my home, but nothing south.

I'm going to file a bug with Apple since turning off location services doesn't clear the cache and see what they say. It does seem to halt the collection of new data though, so anyone specifically worried about this can switch off location services and encrypt their existing backups.

I didn't bother to dig deeper to see if any of this was sent to Apple, mostly since their EULA and response to congress last summer already confirmed they collect other location data on an anonymous basis if people use location services.

Apple formally responds with a FAQ, and a software update will be made available later to do 3 things:

1. reduces the size of the crowd-sourced Wi-Fi hotspot and cell tower database cached on the iPhone,
2. ceases backing up this cache, and
3. deletes this cache entirely when Location Services is turned off.

After personally looking into the files and testing it this weekend, most of what Apple says seems to be plausible, and heres why I believe so:

I think it was an oversight that the cache wasn't being purged, possibly due to last minute iOS 4 changes. Apple has been rushing things lately on the iOS side, and a stressed out engineer could have missed a change he was tasked to make. iOS 3 contained a different cache file called h-cells.plist. iOS 4 (rather 3.2, the iPad only release) changed it to the consolidated.db, and added WiFi. These changes occurred as Apple began using their own WiFi service instead of Skyhook for WiFi and Google for cell tower info. I'm not 100% sure if the previous h-cells.plist was being purged on a regular basis, but from what I've seen, it wasn't as extensive as consolidated.db.

The file being backed up is likely a proper bug/mistake. H-cells.plist wasn't ever backed up, and was stored in /Library/caches/locationd for the root user. The new consolidated.db file is also stored in the same folder path. So where does the bug come in? iOS 4 added a persistent settings file to allow location to be enabled or disabled on an app by app basis. This is controlled by the initial user prompt when using an app, or via a control panel of per app toggles in the system Settings app. iOS 3 and prior would prompt multiple times to use location and lacked a central settings panel. This setting file also gets stored in /Library/caches/locationd. I'd bet that the person who implemented it added the full /Library/caches/locationd folder to the backup include list, instead of his one file. The proper way would have been to put the settings file in the normal place, /Library/Preferences. Definite code/implementation review failure here.

Number three is tied in with Apple claiming it's a bug that turning off Location Services doesn't always disable the cache. Deleting the file will ensure this isn't an issue. In my private testing, turning off Location Services did stop updating the file, but others are reporting it didn't stop for them.

At this point, Apple will still be called to task by the U.S. government as well as several other governments. In the end, I expect they will be required to have some sort of opt-in/opt-out switch on whether any of this data, anonymized or not, is sent back to the mothership. I expect similar requirements for other phone vendors.

The zillion dollar question for me is whether this will ultimately be an opt-in system or an opt-out system. My guess is there will be a united front from the vendors to push for opt-out, and that will probably win in the U.S., but not necessarily other countries.

According to Apple, none of this information is sent from the iPhone TO Apple, and in fact it's downloaded FROM Apple to the iPhone. Which does make sense, because I saw plenty of small dots on my own map, that likely represented WiFi base stations, that my iPhone was never in range of.

This would very much suggest that some location data is sent, in some form, to Apple:

"5. Can Apple locate me based on my geo-tagged Wi-Fi hotspot and cell tower data?
No. This data is sent to Apple in an anonymous and encrypted form. Apple cannot identify the source of this data."

Which would make sense if they are building up their own wifi geo-location database.

and also:

"These calculations are performed live on the iPhone using a crowd-sourced database of Wi-Fi hotspot and cell tower data that is generated by tens of millions of iPhones sending the geo-tagged locations of nearby Wi-Fi hotspots and cell towers in an anonymous and encrypted form to Apple"

I suppose the information does have to go both ways, but the cached data shown on the maps is a slice (albeit one tailored to you) out of Apple's DB.

I'm sure Apple's DB can safely toss out any data even remotely associated with identifying a particular phone. In the end they have a huge map made up of geo-locations for hotspots and cell towers which they're storing, and not having to pay Skyhook for.

I don't have a problem with that. It's the same story for Skyhook and Google. Though I wouldn't be surprised to find out that Google's data includes some identifying bits. Of large companies, I don't trust any less than Google at this point.

I see this as fair, and the process will help educate people a bit more about the issues modern phones raise. And I do think companies, including Apple, need to ensure proper reviews occur with any systems associated with handling location data. This is very similar to the privacy outcry over Buzz, with lots of misinformation, some real information, and good long term changes to ensure a slip-up doesn't occur again.

The iPhone does have an opt-out switch already, (the Location Services setting on the front page of Settings) and I could see this flipping to opt-in at the system level, similar to how all apps face an opt-in by default.

It makes no sense at all -- or at best is a misleading half-truth. The information they're talking about ("all the cell towers in Toronto") is sent from Apple to Iphone. Why is that particular slice of the global database sent? Because the Iphone has sent to Apple a request saying, "send me all the cell towers near <this specific location>". Apple still gets told your whereabouts.

Peter

Exactly. For the life of me I can't understand why so many companies don't get that it's better to lay all your cards on the table when this stuff happens than to hem and haw and obfuscate. If El Steve-o had just come out right away and said "we're collecting stuff, it's not very detailed, we'll put in a mechanism to turn it off without losing any location-aware functionality," this could have blown over before senators started getting involved. See also, Sony on the PSN outage, TEPCO on the scale of the containment problems at Fukushima, etc.

I guess the nature of cover-ups is that we don't hear so much about the successful ones, but it seems to me the risk of letting these questions linger does more harm than any good that comes from hiding the truth.

I get the impression is that is exactly what they have done, ie they have responded just as quickly as they have been able to understand exactly what the situation is.

But they did, last summer (2010), due to a general congressional inquiry about location tracking regarding all smart phones. And even prior to that inquiry, there was information about what Apple was doing, from the EULA, to WWDC sessions and information readily available on the developer site.

There was nothing new discovered with this current issue, beyond bug related problems. The existence of the cache was known (directly from Apple), the fact it was backed up was known (from forensics experts last year and is now known to be a bug), and the fact that it doesn't contain the exact location of the phone. The only thing new about the report from O'Reilly Media was the hype, and an application to incorrectly map the data.


Bruno's statement should probably be clarified. Apple stated the cached file is never sent to Apple. That's what they mean. Last summer, they already acknowledged they receive data from users based on location requests. And they explained that under iOS 3.0, Google and Skyhook also received the requests, but as of iOS 4, only Apple sees it.

As far as the data Apple gets for this exact location issue, they see a request not that a phone is at exact position X, but instead they receive data that says "I can see a cell tower with ID number 53022, and another with ID 53023, send me all cell tower data around those towers". All the triangulation involving power levels of the signals from each tower is done locally on the phone. With WiFi data, Apple receives requests that do narrow down the area a bit more, but these requests may also be based on where the user is searching, and not the location of the device. This is done for the iOS devices lacking cellular radios and GPS chips to still allow them to show a basic location marker.

All Things Digital has an interview with Jobs, Schiller, and Forstall posted. One point seems to be that they think the misunderstanding is an education issue, and that Apple, and other companies need to do a better job at explaining how all these location services work.

Forstall also commented the file is being truncated now, but only when the file hits 2MB. This was clearly high enough to contain a ton of data, and as previously announced, they will be switching to a time based truncation method of a week.

Something revealed earlier in the official announcement and reiterated in the interview is that Apple is planning on using the anonymous data they do collect for a traffic reporting system. No further details on it were given though. This may be part of iOS 5, with the rumored departure from Google Maps to an Apple built maps program.


Yep, thats Steve Jobs. Looks like his health problems haven't impacted his feistiness.

I wonder if they got him to comment on his health off the record. I hope he hangs in there, but realistically he's not been looking very good at all. Statistically, he's already on borrowed time.

I haven't been following every detail of this story, but what evidence has been presented that this is a bug rather than a feature? The only reference I've seen to it being a bug is your speculation to that effect. It very well may be, but I didn't know that had transitioned from your well-educated guess to a verified fact.

That said, I get that much of this was known about by some experts, but once it became known to the larger audience, Apple had a responsibility to respond in a timely manner to it, and to inform users of their plans to correct the problem. In my opinion, they failed on that, as many companies do. This is more of a complaint about how tech companies think they can just weather the storm with vague press releases when user privacy concerns are bringing them bad press, not a specific criticism of Apple alone.

Yes, but those statistics include people of all means. It's quite a marvel that he's doing as well as he apparently is, but less of a marvel when you consider his immense personal fortune he can pour into getting the best medical care available.

Only Apple's claim. So they're either lying or telling the truth.

This part only applies to the data file being backed up to a computer.

Anyone who works deeply with OS X and iOS as far as how the filesystem is laid out, and how Time Machine/iOS backups work would likely come to the same conclusion as my speculation about it being a bug. Some engineer decided to save a preferences file in a cache folder and removed that folder from the backup exclude list. Or, the other possibility I see is that the caches folder was never properly added to the backup exclude list like it should have been.

One mistake I did make in my checks was the location of the data in the past. iOS prior to 3.2 stored the older h-cell.plist cache under /root/Library/Caches, the cache folder for the root user on the phone. 3.2 moved it to the user partition under ~/Library/Caches. I had incorrectly assumed the location was still the same. Knowing the change happened reinforces my belief this is a bug, as the previous plist wasn't part of the backups. Why make a change now, when they previously saw no need to back up caches in the past? From information released at WWDC (that I can't re quote here, I think 2010 sessions are still under NDA), and from a previous link to Alex Levinston's analysis, the changes had to do with supporting the multitasking features of iOS 4. Combine the analysis of file location, changes needed for iOS 4, and Apple's paranoia about location and analytics*, and it seems more like a bug/oversight vs a feature intentionally added.

I get that not everyone is trustworthy all the time, including Jobs. After looking into the issue myself by using my phone to test it directly, along with my experience in the software field as a build engineer, it all comes across to me as a bug or an oversight. I'm sure many developers here have been through those last minute crunches, and something falls through the cracks. In this case looking at Apple's tight release schedules for iOS over the years, it seems to me there were plenty of corners cut to get things out the door. I don't see this as intentional malice, just crunch time carelessness. It does point to a need for stronger code and implementation reviews at Apple, especially when a users privacy is concerned.

The only way this would be a verifiable fact is if the engineer who either implemented this "feature", or didn't implement the changes for iOS 4 properly were to come forward. That is highly unlikely to happen due to his NDA work agreements and such, so I'm not sure how to really convince you beyond what has been said and linked here. Some of my earlier links contain discussions other people are having that have been researching this, and most lean towards a mistake and not malice.

* I spent about 15 minutes trying to find good links to the previous issues Apple discussed last year, but Google is mostly returning results on the current 2011 story. Basically the changes made in iOS 4 to show what apps last gathered any location data came out of Apple discovering 3rd party apps were silently reporting more info then they thought. Some app revealed early details of the iPad when in development. This lead to some analytics lockdowns, and an awareness about apps gathering location details when they shouldn't. Jobs revealed more info on their beliefs regarding location last year at the D8 conference.


From having worked closely with some of the community folks in the games industry, it seemed to be a balancing act. In general, people are going to be posting random crazy things all the time. Some times, those posts blow up into a bigger rumor. A company can't sit there and deny every single rumor thrown at them. Even a simple statement of "we are looking into this rumor" can cause more problems.

To draw a parallel here (and don't get stuck in the details, this is just a general comparison to another recent event), should Obama have responded every time his location of birth was brought up? After all, it is a pretty major thing concerning his eligibility as President. Initially he ignored the situation since he passed all the checks required to be on the ballot in all 50 states. Then later he bowed to pressure and released his short form certificate. Time goes on, the rumors continue to churn and bubble, and eventually blow up again when Trump starts talking about it. Nothing changed, except the hype. And Obama once again bowed to the pressure and released the long form certificate today. Will the release end the issue? Nope, based on all the birther comments still showing up around the web.

Yes, this particular situation is a little different since it involves potential privacy concerns, but this overall issue of location tracking came up last year as a big deal. Apple may have felt that they already addressed these concerns with their EULA, the congressional response, WWDC coverage, and the systems they put into place last year (short form release). More then just a few experts knew about it, but it wasn't a big deal. Nothing really changed between then and now, but for some reason the O'Reilly report (Trump) stirred it up again. Apple's latest response and action is similar to the long form certificate release. Bugs were identified (ignoring the backup, they did confirm the bug where cached data was still being collected with Location Services toggled off), oversights were identified (the 2MB cap on the file was too big), and corrective actions are being taken. Will this quell the general location tracking fears? Probably not.

Ok, you know what? It's clear I was just too close to this all, with my own personal investigation into it. My mind has changed now that I've seen the excellent coverage of whats going on from Next Media Animation
(potentially NSFW video)

I suspect there is a disconnect between what an individual would wish for in terms of, speed of response, and what any corporation can actually muster.

For Apple to investigate what the issue was, and get the response vetted by all the necessary people.... I think they moved pretty fast.


I hope they can get ahead of the misinformation.

Been running 4.3.3 (the version that cuts the cache down to 7 days), and I'm definitely noticing the change. Went to lunch today to an area I haven't been in for about 2 weeks now, and Maps initially showed an estimate location circle the size of the entire Orange County area. It took a good 2-3 second for it to narrow it down closer, due to the network lag of having to pull the data over the network. Never saw a guess that large on the initial load since iOS 4.0.

Also, if anyone is interested in the senate committee hearing about mobile data privacy that Franken called, This is my Next had a good liveblog of it. It went beyond just location tracking and also talked about other data mobile apps appear to be collecting.

I found two things interesting from it, the FTC has a lab where they test various apps on different platforms and snoop the traffic to see what is being sent out, and:

The engineers at Google were busy engineering. They pay lobbyists to be lobbyists.