{ "cells": [ { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "%reload_ext autoreload\n", "\n", "import os\n", "import zipfile\n", "import wget\n", "import networkx as nx\n", "from main import *\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Discovering the datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To perform our analysis, we will use the following datasets:\n", "\n", "- **Brightkite**\n", "- **Gowalla**\n", "- **Foursquare**\n", "\n", "We can download the datasets using the function `download_dataset` from the `utils` module. It will download the datasets in the `data` folder, organized in sub-folders in the following way:\n", "\n", "```\n", "data/\n", "├── brightkite\n", "│   ├── loc-brightkite_edges.txt.gz\n", "│   ├── loc-brightkite_totalCheckins.txt.gz\n", "├── foursquare\n", "│   ├── loc-gowalla_edges.txt.gz\n", "│   ├── loc-gowalla_totalCheckins.txt.gz\n", "└── gowalla\n", " ├── dataset_ubicomp2013_checkins.txt\n", " ├── dataset_ubicomp2013_tags.txt\n", " └── dataset_ubicomp2013_tips.txt\n", "```\n", "\n", "If any of the datasets is already downloaded, it will not be downloaded again. For futher details about the function below, please refer to the `utils` module." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The brightkite dataset is already downloaded and extracted as .txt file, if you want to download again the .gz file with this function, delete the .txt files in the folder\n", "The gowalla dataset is already downloaded and extracted as .txt file, if you want to download again the .gz file with this function, delete the .txt files in the folder\n", "The foursquare dataset is already downloaded and extracted as .txt file, if you want to download again the .gz file with this function, delete the .txt files in the folder\n" ] } ], "source": [ "download_datasets()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's have a deeper look at them.\n", "\n", "## Brightkite\n", "\n", "[Brightkite](http://www.brightkite.com/) was once a location-based social networking service provider where users shared their locations by checking-in. The friendship network was collected using their public API. The network was originally directed but the authors of the dataset have constructed a network with undirected edges when there is a friendship in both ways. They also have also collected a total of `4491143` checking of these users over the period of Apr. 2008 - Oct. 2010.\n", "\n", "Here is an example of check-in information" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
usercheck-in timelatitudelongitudelocation_id
002010-10-17T01:48:53Z39.747652-104.99251088c46bf20db295831bd2d1718ad7e6f5
102010-10-16T06:02:04Z39.891383-105.0708147a0f88982aa015062b95e3b4843f9ca2
202010-10-16T03:48:54Z39.891077-105.068532dd7cd3d264c2d063832db506fba8bf79
302010-10-14T18:25:51Z39.750469-104.9990739848afcc62e500a01cf6fbf24b797732f8963683
402010-10-14T00:21:47Z39.752713-104.9963372ef143e12038c870038df53e0478cefc
\n", "
" ], "text/plain": [ " user check-in time latitude longitude \\\n", "0 0 2010-10-17T01:48:53Z 39.747652 -104.992510 \n", "1 0 2010-10-16T06:02:04Z 39.891383 -105.070814 \n", "2 0 2010-10-16T03:48:54Z 39.891077 -105.068532 \n", "3 0 2010-10-14T18:25:51Z 39.750469 -104.999073 \n", "4 0 2010-10-14T00:21:47Z 39.752713 -104.996337 \n", "\n", " location_id \n", "0 88c46bf20db295831bd2d1718ad7e6f5 \n", "1 7a0f88982aa015062b95e3b4843f9ca2 \n", "2 dd7cd3d264c2d063832db506fba8bf79 \n", "3 9848afcc62e500a01cf6fbf24b797732f8963683 \n", "4 2ef143e12038c870038df53e0478cefc " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "brightkite_path = os.path.join(\"data\", \"brightkite\", \"loc-brightkite_totalCheckins.txt\")\n", "Brightkite_df = pd.read_csv(brightkite_path, sep=\"\\t\", header=None, names=[\"user\", \"check-in time\", \"latitude\", \"longitude\", \"location_id\"])\n", "\n", "Brightkite_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Gowalla\n", "\n", "Gowalla is a location-based social networking website where users share their locations by checking-in. The friendship network is undirected and was collected using their public API. The authors have collected a total of `6442890` check-ins of these users over the period of Feb. 2009 - Oct. 2010.\n", "\n", "Here is an example of check-in information" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
usercheck-in timelatitudelongitudelocation_id
002010-10-19T23:55:27Z30.235909-97.79514022847
102010-10-18T22:17:43Z30.269103-97.749395420315
202010-10-17T23:42:03Z30.255731-97.763386316637
302010-10-17T19:26:05Z30.263418-97.75759716516
402010-10-16T18:50:42Z30.274292-97.7405235535878
\n", "
" ], "text/plain": [ " user check-in time latitude longitude location_id\n", "0 0 2010-10-19T23:55:27Z 30.235909 -97.795140 22847\n", "1 0 2010-10-18T22:17:43Z 30.269103 -97.749395 420315\n", "2 0 2010-10-17T23:42:03Z 30.255731 -97.763386 316637\n", "3 0 2010-10-17T19:26:05Z 30.263418 -97.757597 16516\n", "4 0 2010-10-16T18:50:42Z 30.274292 -97.740523 5535878" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "gowalla_path = os.path.join(\"data\", \"gowalla\", \"loc-gowalla_totalCheckins.txt\")\n", "\n", "Gowalla_df = pd.read_csv(gowalla_path, sep=\"\\t\", header=None, names=[\"user\", \"check-in time\", \"latitude\", \"longitude\", \"location_id\"])\n", "\n", "Gowalla_df.head() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Foursquare\n", "\n", "[Foursquare](https://foursquare.com/) is a location-based social networking website where users share their locations by checking-in. This dataset includes long-term (about 10 months) check-in data in New York city and Tokyo collected from Foursquare from 12 April 2012 to 16 February 2013. It contains two files in tsv format. Each file contains 8 columns, which are:\n", "\n", "1. User ID (anonymized)\n", "2. Venue ID (Foursquare)\n", "3. Venue category ID (Foursquare)\n", "4. Venue category name (Foursquare)\n", "5. Latitude\n", "6. Longitude\n", "7. Timezone offset in minutes (The offset in minutes between when this check-in occurred and the same time in UTC)\n", "8. UTC time\n", "\n", "Here is an example of check-in information from the New York dataset:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
UserIDVenueIDCategoryIDCategoryNameLatitudeLongitudeTimezone offset in minutesUTC time
047049bbd6c0f964a520f4531fe34bf58dd8d48988d127951735Arts & Crafts Store40.719810-74.002581-240Tue Apr 03 18:00:09 +0000 2012
19794a43c0aef964a520c6a61fe34bf58dd8d48988d1df941735Bridge40.606800-74.044170-240Tue Apr 03 18:00:25 +0000 2012
2694c5cc7b485a1e21e00d357114bf58dd8d48988d103941735Home (private)40.716162-73.883070-240Tue Apr 03 18:02:24 +0000 2012
33954bc7086715a7ef3bef9878da4bf58dd8d48988d104941735Medical Center40.745164-73.982519-240Tue Apr 03 18:02:41 +0000 2012
4874cf2c5321d18a143951b5cec4bf58dd8d48988d1cb941735Food Truck40.740104-73.989658-240Tue Apr 03 18:03:00 +0000 2012
\n", "
" ], "text/plain": [ " UserID VenueID CategoryID \\\n", "0 470 49bbd6c0f964a520f4531fe3 4bf58dd8d48988d127951735 \n", "1 979 4a43c0aef964a520c6a61fe3 4bf58dd8d48988d1df941735 \n", "2 69 4c5cc7b485a1e21e00d35711 4bf58dd8d48988d103941735 \n", "3 395 4bc7086715a7ef3bef9878da 4bf58dd8d48988d104941735 \n", "4 87 4cf2c5321d18a143951b5cec 4bf58dd8d48988d1cb941735 \n", "\n", " CategoryName Latitude Longitude Timezone offset in minutes \\\n", "0 Arts & Crafts Store 40.719810 -74.002581 -240 \n", "1 Bridge 40.606800 -74.044170 -240 \n", "2 Home (private) 40.716162 -73.883070 -240 \n", "3 Medical Center 40.745164 -73.982519 -240 \n", "4 Food Truck 40.740104 -73.989658 -240 \n", "\n", " UTC time \n", "0 Tue Apr 03 18:00:09 +0000 2012 \n", "1 Tue Apr 03 18:00:25 +0000 2012 \n", "2 Tue Apr 03 18:02:24 +0000 2012 \n", "3 Tue Apr 03 18:02:41 +0000 2012 \n", "4 Tue Apr 03 18:03:00 +0000 2012 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "foursquare_NYC_path = ny = os.path.join(\"data\", \"foursquare\", \"dataset_TSMC2014_NYC.txt\")\n", "foursquare_TKY_path = ny = os.path.join(\"data\", \"foursquare\", \"dataset_TSMC2014_TKY.txt\")\n", "\n", "foursquare_NYC_df = pd.read_csv(foursquare_NYC_path, sep=\"\\t\", header=None, names=[\"UserID\", \"VenueID\", \"CategoryID\", \"CategoryName\", \"Latitude\", \"Longitude\", \"Timezone offset in minutes\", \"UTC time\"], encoding=\"utf-8\", encoding_errors=\"ignore\")\n", "\n", "foursquare_NYC_df.head()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# remove from memory, they were created only for aesthetic purposes in the notebook\n", "\n", "del Brightkite_df\n", "del Gowalla_df\n", "del foursquare_NYC_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Building the networks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are asked to construct the networks for the three datasets as un undirected grah $M = (V, E)$, where $V$ is the set of nodes and $E$ is the set of edges. The nodes represent the users and the edges indicates that two individuals visited the same location at least once.\n", "\n", "We can use the fucntion create_graph from the `utils` module to create the networks. It takes as input the path to an edge list file and returns a networkx graph object. For further details about the function below, please refer to the `utils` module." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of nodes added to the graph brightkite: 51406\n" ] }, { "ename": "KeyboardInterrupt", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", "\u001b[0;32m/tmp/ipykernel_51637/2618808480.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mBrightkite_G\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_checkins_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"brightkite\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mGowalla_G\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_checkins_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"gowalla\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mFoursquare_G\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_checkins_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"foursquareNYC\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/github/small-worlds/main.py\u001b[0m in \u001b[0;36mcreate_checkins_graph\u001b[0;34m(dataset)\u001b[0m\n\u001b[1;32m 132\u001b[0m \u001b[0;31m# now add the edges, try to use pandas to speed up the process\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 133\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0muser1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muser2\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mcombinations\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0musers\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m \u001b[0mintersection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0musers_venues\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0muser1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0musers_venues\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0muser2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 135\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mintersection\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 136\u001b[0m \u001b[0mG\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_edge\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muser1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muser2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mintersection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mKeyboardInterrupt\u001b[0m: " ] } ], "source": [ "Brightkite_G = create_checkins_graph(\"brightkite\")\n", "Gowalla_G = create_checkins_graph(\"gowalla\")\n", "Foursquare_G = create_checkins_graph(\"foursquareNYC\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can have a look at the number of nodes and edges in each network." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datasetnodesedges
0brightkite58228214078
1gowalla196591950327
2foursquare1083282405
\n", "
" ], "text/plain": [ " dataset nodes edges\n", "0 brightkite 58228 214078\n", "1 gowalla 196591 950327\n", "2 foursquare 1083 282405" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset = [\"brightkite\", \"gowalla\", \"foursquare\"]\n", "nodes = [len(Brightkite_G.nodes()), len(Gowalla_G.nodes()), len(Foursquare_G.nodes())]\n", "edges = [len(Brightkite_G.edges()), len(Gowalla_G.edges()), len(Foursquare_G.edges())]\n", "\n", "df = pd.DataFrame({\"dataset\": dataset, \"nodes\": nodes, \"edges\": edges})\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, the foursquare dataset has a very small number of nodes. Even tho it has 227428 check-ins, the unique users (the nodes) are only 1083. The Tokyo dataset is about 2 times bigger, with 537703 check-ins and 2294 nodes. Since we are in the same order of magnitude, we will focus on the New York dataset, in the style of a classic Hollywood movie about aliens invasions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Analysis of the structure of the networks" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "path = \"data/brightkite/loc-brightkite_totalCheckins.txt\"\n", "# modify the file, take only the first and last column, return a test.txt file. Use pandas\n", "\n", "def modify_file(path):\n", " df = pd.read_csv(path, sep=\"\\t\", header=None, names=[\"user\", \"check-in time\", \"latitude\", \"longitude\", \"location_id\"])\n", " df = df.iloc[:, [0, 4]]\n", " df.to_csv(\"test.txt\", sep=\"\\t\", header=None, index=None)\n", "\n", "modify_file(path)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.10.6 64-bit", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1" } } }, "nbformat": 4, "nbformat_minor": 2 }