I’ll admit. I don’t really follow video games that closely. Mostly owing to the fact that I’m frankly too busy with other things to fill my time with them these days. It’s not that I shun them for any particular reason. I used to be a fairly big gamer, but those days are long gone.
So then, I was surprised this morning by a bunch of articles on the (un)availability of the services tasked with keeping hordes of Pokemon Go players happy this past week.
I have a 6yr old daughter who is infatuated with Pokemon these days thanks to Netflix, but I honestly don’t have a clue what they are, or why anyone would like them. I’ll let you come to your own opinions on the merits of the game, but it does have a pretty interesting history for us technical folks. To save you some reading, Niantic is a startup spun out of Google in 2015, birthed from John Hanke, father of Keyhole, which was aquired by Google eventually became Google Earth.
During a work conversation today, a question came up that I thought for sure would be easy to answer. If Niantic’s services are being crushed by load, where are they hosted? I figured a quick google search would unearth the answer to this query, but alas my Google foo was not up to the task. So let’s dig in, shall we?
Pokemon Go has recently been released on Android and iOS. I an iOS user, so it’s not quite as easy as simply running a tcpdump on my laptop while running the app and finding the source (unless you’re jailbroken and whatnot). There will be a myriad of ways to do this by the way, but I thought documenting one of the ways to grab this data might be interesting as I learned a few things along the way as well so why not share it.
First things first. Let’s grab a .pcap of the data coming off my iOS device so we can analyse our outbound traffic and find out were it’s headed.
A quick check with tcpdump on en0 interface ( my wireless NIC ) was turning up only broadcast/multicast traffic only, but not traffic which I expected should be in flight in the air which I expected to see. After all, wireless IS shared media, right?
barnesry-mbp:~ barnesry$ ifconfig en0 en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 ether 60:f8:1d:c4:3b:38 inet6 fe80::62f8:1dff:fec4:3b38%en0 prefixlen 64 scopeid 0x4 inet 10.0.1.10 netmask 0xffffff00 broadcast 10.0.1.255 nd6 options=1<PERFORMNUD> media: autoselect status: active barnesry-mbp:$ sudo tcpdump -i en0 host 10.0.1.3 Password: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes 23:16:16.338903 IP 10.0.1.3.mdns > 220.127.116.11.mdns: 0 [2q] [1au] PTR (QU)? _airplay._tcp.local. PTR (QU)? _raop._tcp.local. (78) 23:16:16.645533 ARP, Request who-has 10.0.1.3 tell 10.0.1.13, length 28
What then to do?
If you want to sniff all the wireless traffic around, you’ll need to drop your wireless card into ‘monitor’ mode. This is also outlined in the tcpdump man page in OSX and will result in passing up everything from the 802.11 layer including wireless SNR, etc but makes the capture much more interesting!
-I Put the interface in “monitor mode”; this is supported only on IEEE 802.11
Wi-Fi interfaces, and supported only on some operating systems.
barnesry-mbp:~ barnesry$ sudo tcpdump -I -i en0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on en0, link-type IEEE802_11_RADIO (802.11 plus radiotap header), capture size 65535 bytes 23:09:30.065941 4111060402us tsft 1.0 Mb/s 2452 MHz 11g -75dB signal -92dB noise antenna 0 Beacon (CenturyLink6989) [1.0* 2.0* 5.5* 11.0* 18.0 24.0 36.0 54.0 Mbit] ESS CH: 9, PRIVACY 23:09:30.075347 4111071507us tsft 1.0 Mb/s 2452 MHz 11g -92dB noise antenna 0 23:09:30.075625 4111071935us tsft 1.0 Mb/s 2452 MHz 11g -92dB noise antenna 0 Acknowledgment RA:b8:c7:5d:12:8e:f5 (oui Unknown) 23:09:30.112349 4111106996us tsft 1.0 Mb/s 2452 MHz 11g -92dB noise antenna 0 Beacon (Goiter) [1.0* 2.0* 5.5* 11.0* 6.0 9.0 12.0 18.0 Mbit] ESS CH: 9, PRIVACY 23:09:30.168411 4111162822us tsft 1.0 Mb/s 2452 MHz 11g -76dB signal -92dB noise antenna 0 Beacon (CenturyLink6989) [1.0* 2.0* 5.5* 11.0* 18.0 24.0 36.0 54.0 Mbit] ESS CH: 9, PRIVACY 23:09:30.214709 4111209396us tsft 1.0 Mb/s 2452 MHz 11g -38dB signal -92dB noise antenna 0 Beacon (Goiter) [1.0* 2.0* 5.5* 11.0* 6.0 9.0 12.0 18.0 Mbit] ESS CH: 9, PRIVACY 23:09:30.270764 4111265204us tsft 1.0 Mb/s 2452 MHz 11g -73dB signal -92dB noise antenna 0 Beacon (CenturyLink6989) [1.0* 2.0* 5.5* 11.0* 18.0 24.0 36.0 54.0 Mbit] ESS CH: 9, PRIVACY 23:09:30.284515 4111280917us tsft short preamble 24.0 Mb/s 2452 MHz 11g -41dB signal -92dB noise antenna 0 Clear-To-Send RA:60:f8:1d:c4:3b:38 (oui Unknown)
What I also didn’t know is there’s a built in wireless diagnostic tool in OSX which can help with this and the process is documented HERE. This will result in a .wpcap file dropped on your desktop containing all wireless traffic snorted off your wifi card. Cool.
I closed all the applications on my phone, and turned it off prior to starting the capture. We will need the initial handshake captured when the phone associates to the AP in order to decode the traffic later.
So we start the wireless capture (and my timer) and turn the phone on.
Within about 30 seconds, the phone is on. It’s likely there will a bunch of stuff happening on boot, so we’ll wait a couple minutes for everything to settle down before launching anything which will make triangulation of our game traffic much easier. Then we’ll log in, move around a bit, then close down the capture.
Let’s load that resulting .wpcap into Wireshark for a look.
This is interesting… we’ve got data, but we can’t read much of it. That’s because it’s encrypted with WPA2-PSK (PSK=pre-shared-key) or WPA2-Personal if you’re my Airport Extreme AP. So we’ll need to decrypt the data to get anything useful here. We’ll do that in a bit.
First, let’s narrow our search down a bit, we’ll need the MAC address of my iPhone. We can get this by navigating to Settings–>General–>About–>Wifi Address on the phone. In my case the MAC address ends in 14:18 which should be good enough to filter on.
Rather than pick through the capture looking for this, or using a display filter let’s use some of the built in tools Wireshark provides to help us.
Navigate to Statistics–>WLAN Traffic.
This will open a window similar to below. I filtered on Ch. 9 to find my AP (Goiter – I name my APs after human afflictions) which then displays the packet statistics by MAC address for my AP. Cool right? Remember the last two octets 14:18 from my phone? It’s responsible for 31.56% of the traffic on this AP.
Now – to filter traffic to ONLY my iPhone, let’s apply a filter to the capture. Right click on the MAC, and choose “Apply As Filter”
Now we’re only looking at filtered data specific to my phone, but now we need to decode it. You can do this as outlined in the Wireshark Wiki (since you already know your WPA passphrase, right?)
In my case, I chose Edit–>Preferences, then Select Protocols and browse down the IEEE 802.11. From here, ensure Enable Decryption is checked, and Edit your keys.
Here I used wpa-pwd and simply entered my SSID passphrase in cleartext (I erased it for this screenshot).
Alternatively, I believe you can also select wpa-psk but you’ll need to enter in the full 64 digit hex key, as outlined here. The cleartext version worked for me so I didn’t bother with the latter.
Now, we’ve got a full decrypted traffic stream we can work with. You’ll also notice in the bottom status bar I’ve captured 350k odd packets, and I’m displaying only 70k of them (my iPhone) so let’s chuck the rest as we don’t need to work with such a big file. Choose File–>Export Specified Packets, and write our file out containing only the Displayed packets we’re interested in.Then re-open our smaller (more specific) file.
Time for some analysis. Choose Statistics–>Conversations. This will net us a table of each pair of IP endpoints, so we can get a better view of how much data was flowing between pairs of machines during the capture. I found sorting by Rel Start to be useful, as I could match the appearance of certain IP addresses against my testing timeline. (Remember I waited for 120 or so seconds to start my test?)
Excellent. There’s a notable gap in activity between 75s and 171s, so let’s start there and assume we probably kicked things off around 171 seconds into the capture.
For a high level view of what’s going on, DNS is a good place to start so let’s filter for that in the display filter using dns.
Some interesting stuff to glean here. It looks like our first call is at 171s into the capture as well, attempting to resolve pgorelease.nianticlabs.com. Bingo.
This very likely confirms our relative start time. There’s some other queries for upsight and kontagent, which a quick google search will reveal are mobile analytics companies, likely in play here as well providing data back to the publisher. Next we have accounts.google.com which is also explainable as I logged in using my Google account. The interesting stuff ends around 209s when I snapped a picture on my phone, and it immediately went to upload it to iCloud. More on that in a bit. 🙂
Here’s the key IP pairs (I think) we should be interested in.
- storage.googleapis.com -> which CNAMES to 18.104.22.168 (Whois 1E100net. aka. google CDN)
- cl3.apple.com.edgekey.net -> which CNAMES to akamai CDN
Let’s see if our traffic profiles confirm some of this. Navigate to Statistics–>Conversations. I’ve selected IPv4 here, and sorted by Rel Start which should allow us to rule out any traffic occurring before relative timestamp 171sec.
Since our game should be pretty chatty over time as we load the game, and report location, and receive game status I’d expect a high packet count, but perhaps low byte count. (small packets, but lots of them). We’ve actually got four good candidates here.
- 22.214.171.124 – google 1e100.net (Denver)
- 126.96.36.199 – google 1e100.net (Seattle)
- 188.8.131.52 – googleusercontent.com
- 184.108.40.206 – Akamai via cl3.apple.com.edgekey.net
To resolve IP addresses to owners as I’ve done above, we’ll need to perform some WhoIs lookups to see who owns some of this IP space.
For this I personally like to use network-tools.com which has a good suite of tools, available in a consumable format. You can also use your trusty command line tools as well to accomplish this. The example below came from konagent.net, who apparently host with Softlayer.
barnesry-mbp:~ barnesry$ whois -h whois.radb.net 220.127.116.11 route: 18.104.22.168/18 descr: customer Alestra origin: AS32098 mnt-by: MAINT-AS32098 changed: firstname.lastname@example.org 20150618 #18:44:54Z source: RADB route: 22.214.171.124/18 descr: auto-generated route object for 126.96.36.199/18 origin: AS11172 mnt-by: MAINT-AS11172 changed: email@example.com 20141031 #09:44:53Z source: RADB route: 188.8.131.52/18 descr: SOFTLAYER-sjc01 origin: AS36351 notify: firstname.lastname@example.org mnt-by: MAINT-AS36351 changed: email@example.com 20110104 source: RADB route: 184.108.40.206/18 descr: REACH (Customer Route) tech-c: RRNOC1-REACH origin: AS36351 remarks: This auto-generated route object was created remarks: for a REACH customer route remarks: remarks: This route object was created because remarks: some REACH peers filter based on these objects remarks: and this route may be rejected remarks: if this object is not created. remarks: remarks: Please contact firstname.lastname@example.org if you have any remarks: questions regarding this object. notify: email@example.com mnt-by: MAINT-REACH-NOC changed: firstname.lastname@example.org 20140206 source: REACH
Since Niantic spun out of Google in 2015, it’s a pretty good guess they’re hosted in Google Compute Engine of which their two IP’s listed above account for ~700pkts (or about 1/2 of the traffic from the big four) listed earlier.
I think we can further discount the IP address 220.127.116.11 as some background Apple stuff hosted in Akamai perhaps related to game launch (validation and such). This leaves… Google Compute Engine.
We can also rule out the large byte transfer at Rel Start 225.647351 as a deep dive on that lines up with our DNS query for Amazon AWS Oregon S3 bucket, preceded by some chatter to Apple CDN servers. This is further confirmed above by a whois on 18.104.22.168 (AWS-S3) and also by the upload ratio for this conversation… stuff going up, up, up into the cloud when I snapped my picture on my iPhone.
The intent of this post was initially to solve my earlier question of where Niantic was hosting Pokemon Go, which perhaps could have easier been assumed simply by looking at where Niantic was birthed (Google), but where is the fun in that?
The end result is less a proof point of this fact, but more a view into various troubleshooting tools and methodologies available to really deep dive into various network traffic patterns on the internet.
If you’ve gotten this far, hopefully it was an interesting enough read.