Since the beginning of 2016, I've been learning to write code in various languages while exploring many common architectures for machine learning. I've been working on some personal projects and want to share one of them with you! Recently, a dataset of Uber pickups in New York in 2014 was posted on Kaggle. The data was posted as columns of latitudes, longitudes and date/time for each pickup. I used that dataset to create this animation.
For those who are curious, here's how I created the animation (click here for the code):
The data was converted to the proper formats (such as datetime64 format for the date/time) and Google Maps was used to get the max and min latitudes and longitudes (lat/long) of Manhattan Island and removed pickups that happened outside of this zone.
A 3-dimensional array of numbers was produced using the min and max lat/long for the height and width and time for the depth. The height and width of the array represent each frame of the animation and I allowed the user the determine how wide each pixel represents in the real-world, otherwise defaulting to 100 meters, and how long for each frame to be, otherwise defaulting to 30 minutes.
This way, every row, column and depth position has its own index value and I could use matrix multiplication (which computers are really good at) to almost instantly convert the values of the dataset into values representing index positions (saving LOTS of processing). This way, instead of a pickup being labeled as "40.702 Lat, -73.955 Long, April 12th at 11:23", it became labeled as "74th row, 63rd column, 923rd time point".
Then, it was important to define how each pickup would be represented inside the array. Was it going to be a single pixel? Was there going to be a region bright in the center and dimmer around the edges? While the data would suggest that there should only be a single point of activation from each pickup, I propose that this result reflects the way the pickup was measured, rather than the reality. The reality is more psychologically complex: when someone is asking to be picked up at a certain location and time, they would still be willing to be picked up a little distance and time away from the specified location and time. Therefore, our model requires a decay constant for distance and time. Alpha and sigma values are introduced that ask the user for the time and distance to 63% decay point (assuming logarithmic decay), defaulting to 30 minutes and 100 meters if left unspecified. The decay is calculated up to a minimum decay were the influence of the pickup is too small to care.
Using our decay model, a cube is generated that represents the way a single pickup influences the 3D array, with a value of 1 at the very center and smaller values with distance and time. This cube is then added to our 3D array at every index position we defined earlier.
Then, the length and width of the array are animated over the depth to produce the above video without the map overlay.
To create the overlay, I used a little Photoshop, and Premier magic after pulling maps of Manhattan Island off of Google Maps using the min and max lat/long I used earlier to exclude irrelevant pickups.
The above video was created with each pixel representing 50 meters and each frame representing 15 minutes. Sigma of 50 meters and alpha of 15 minutes were used as well, up to a minimum decay of 0.02.
My next step if to use the array here to predict future Uber pickups. From the video, it's possible to determine a regular 'breathing' pattern that I'm hoping machine learning can extract and use to project. Stay tuned!