-Thisfileisnotemeanttoberun,it's just a collection of functions that are used in the other files. It'sjustawaytokeepthecodecleanandorganized.
-WhydoIuseos.path.joinandnotthe"/"?Becauseit's more portable, it works on every OS, while "/" works only on Linux and Mac. If you want to use it on Windows, you have to change all the "/" with "\". With os.path.join you don'thavetoworryaboutitand,asalways,f***Microsoft.
Thisfunctiontakesininputatsvfilewithtwocolumns,Eachlineinthefileisanedge.Thefunctionreturnsanundirectednetworkxgraphobject.Itusespandastoreadthefilesinceit's faster than the standard python open() function. If we don'twanttousethestandardpythonopen()function,thefollowingcodeworksaswell:
# we are given a .txt in tsv format, with 8 colums. Read the file with pandas, the first two colums are colles "UserID" and "VenueID", the other 6 are useless. Then create a graph with networkx for this function. The unique users ID are the nodes, two nodes are linked, if they have been in the same venue at least once. The weight of the edge is the number of times they have been in the same venue.
Differentlyfromthefunctioncreate_graphusedforthebrightkiteandgowalladataset,wearenotgivenalistofedges,sowecan't use the function nx.from_pandas_edgelist. We have to create the graph manually.
Firstly,weretrivetheuniqueuserIDusingtheset()datastructure:thisarethenodesofourgraph.Sincewedon't want to work with adjacency matrices due to their O(mn) space complexity (even tho, we could memorize them in a compressed way thanks to their sparsity propriety), we use an adjacency list representation of the graph. We create a dictionary with the users ID as keys and the venues ID as values. Two users are connected if they have visited the same venue at least once. The weight of the edge is the number of common venues.
df=pd.read_csv(file,sep="\t",header=None,names=["UserID","VenueID","CategoryID","CategoryName","Latitude","Longitude","Timezone offset in minutes","UTC time"])
df=pd.read_csv(file,sep="\t",header=None,names=["UserID","VenueID","CategoryID","CategoryName","Latitude","Longitude","Timezone offset in minutes","UTC time"],encoding="utf-8",encoding_errors="ignore")
# use the set() data structure to get the unique users ID
users=set(df["UserID"])
users=set(df["UserID"])# get the unique users ID
G=nx.Graph()
G.add_nodes_from(users)
# create a dictionary with the users ID as keys and the venues ID as values
users_venues={}
users_venues={}# key: user ID, value: set of venues ID
"The brightkite dataset is already downloaded and extracted as .txt file, if you want to download again the .gz file with this function, delete the .txt files in the folder\n",
"The gowalla dataset is already downloaded and extracted as .txt file, if you want to download again the .gz file with this function, delete the .txt files in the folder\n",
"Downloading foursquare dataset...\n",
"Download completed of foursquare dataset\n"
"The foursquare dataset is already downloaded and extracted as .txt file, if you want to download again the .gz file with this function, delete the .txt files in the folder\n"
"[Foursquare](https://foursquare.com/) is a location-based social networking website where users share their locations by checking-in. This dataset includes long-term (about 10 months) check-in data in New York city and Tokyo collected from Foursquare from 12 April 2012 to 16 February 2013. It contains two files in tsv format. Each file contains 8 columns, which are:\n",
"\n",
"1. User ID (anonymized)\n",
"2. Venue ID (Foursquare)\n",
"3. Venue category ID (Foursquare)\n",
"4. Venue category name (Foursquare)\n",
"5. Latitude\n",
"6. Longitude\n",
"7. Timezone offset in minutes (The offset in minutes between when this check-in occurred and the same time in UTC)\n",
"8. UTC time\n",
"\n",
"Here is an example of check-in information from the New York dataset:"
"We are asked to construct the networks for the three datasets as un undirected grah $M = (V, E)$, where $V$ is the set of nodes and $E$ is the set of edges. The nodes represent the users and the edges indicates that two individuals visited the same location at least once.\n",
"\n",
"We can use the fucntion create_graph from the utils module to create the networks. It takes as input the path to an edge list file and returns a networkx graph object. For further details about the function below, please refer to the `utils` module."
"We can use the fucntion create_graph from the `utils` module to create the networks. It takes as input the path to an edge list file and returns a networkx graph object. For further details about the function below, please refer to the `utils` module."
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 7,
"metadata": {},
"outputs": [
{
"ename": "UnicodeDecodeError",
"evalue": "'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_tokens\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_with_dtype\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._string_convert\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers._string_box_utf8\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data",
"\nDuring handling of the above exception, another exception occurred:\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.read_low_memory\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._read_rows\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_column_data\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_tokens\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_with_dtype\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._string_convert\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32m~/.local/lib/python3.10/site-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers._string_box_utf8\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mUnicodeDecodeError\u001b[0m: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data"
]
}
],
"outputs": [],
"source": [
"Brightkite_G = create_graph(\"brightkite\")\n",
"Gowalla_G = create_graph(\"gowalla\")\n",
@ -221,14 +503,123 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>dataset</th>\n",
" <th>nodes</th>\n",
" <th>edges</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>brightkite</td>\n",
" <td>58228</td>\n",
" <td>214078</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>gowalla</td>\n",
" <td>196591</td>\n",
" <td>950327</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>foursquare</td>\n",
" <td>1083</td>\n",
" <td>282405</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" dataset nodes edges\n",
"0 brightkite 58228 214078\n",
"1 gowalla 196591 950327\n",
"2 foursquare 1083 282405"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print(\"Brightkite graph has {} nodes and {} edges\".format(Brightkite_G.number_of_nodes(), Brightkite_G.number_of_edges()))\n",
"As we can see, the foursquare dataset has a very small number of nodes. Even tho it has 227428 check-ins, the unique users (the nodes) are only 1083. The Tokyo dataset is about 2 times bigger, with 537703 check-ins and 2294 nodes. Since we are in the same order of magnitude, we will focus on the New York dataset, in the style of a classic Hollywood movie about aliens invasions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Analysis of the structure of the networks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given a social network, which of its nodes are more central? This question has been asked many times in sociology, psychology and computer science, and a whole plethora of centrality measures (a.k.a. centrality indices, or rankings) were proposed to account for the importance of the nodes of a network. \n",
"\n",
"These networks, typically generated directly or indirectly by human activity and interaction (and therefore hereafter dubbed social”), appear in a large variety of contexts and often exhibit a surprisingly similar structure. One of the most important notions that researchers have been trying to capture in such networks is “node centrality”: ideally, every node (often representing an individual) has some degree of influence or importance within the social domain under consideration, and one expects such importance to surface in the structure of the social network; centrality is a quantitative measure that aims at\n",
"revealing the importance of a node.\n",
"\n",
"Among the types of centrality that have been considered in the literature, many have to do with distances between nodes. Take, for instance, a node in an undirected connected network: if the sum of distances to all other nodes is large, the node under consideration is peripheral; this is the starting point to define Bavelas’s closeness centrality [cite], which is the reciprocal of peripherality (i.e., the reciprocal of the sum of distances to all other nodes).\n",
"\n",
"The role played by shortest paths is justified by one of the most well-known features of complex networks, the so-called small-world phenomenon. A small-world network [cite] is a graph where the average distance between nodes is logarithmic in the size of the network, whereas the clustering coefficient is larger (that is, neighborhoods tend to be denser) than in a random Erdős-Rényi graph with the same size and average distance. The fact that social networks (whether electronically mediated or not) exhibit the small-world property is known at least since Milgram’s famous experiment [cite] and is arguably the most popular of all features of complex networks. For instance, the average distance of the Facebook graph was recently established to be just 4.74\n",
"\n",
"## Definitions and conventions\n",
"\n",
"From now on, we consider directed graphs defined by a set $N$ of $n$ nodes and $A \\subseteq N \\times N$ of arcs. We write\n",
"\n",
"For this project, we will focus on the following centrality measures:\n",
Given a social network, which of its nodes are more central? This question has been asked many times in sociology, psychology and computer science, and a whole plethora of centrality measures (a.k.a. centrality indices, or rankings) were proposed to account for the importance of the nodes of a network. \s
\nd These networks, typically generated directly or indirectly by human activity and interaction (and therefore hereafter dubbed social”), appear in a large variety of contexts and often exhibit a surprisingly similar structure. One of the most important notions that researchers have been trying to capture in such networks is “node centrality”: ideally, every node (often representing an individual) has some degree of influence or importance within the social domain under consideration, and one expects such importance to surface in the structure of the social network; centrality is a quantitative measure that aims at revealing the importance of a node. \s
\nd Among the types of centrality that have been considered in the literature, many have to do with distances between nodes. Take, for instance, a node in an undirected connected network: if the sum of distances to all other nodes is large, the node under consideration is peripheral; this is the starting point to define Bavelas's closeness centrality \cite{closeness}, which is the reciprocal of peripherality (i.e., the reciprocal of the sum of distances to all other nodes). \s
\nd The role played by shortest paths is justified by one of the most well-known features of complex networks, the so-called small-world phenomenon. A small-world network \cite{cohen_havlin_2010} is a graph where the average distance between nodes is logarithmic in the size of the network, whereas the clustering coefficient is larger (that is, neighborhoods tend to be denser) than in a random Erdős-Rényi graph with the same size and average distance. The fact that social networks (whether electronically mediated or not) exhibit the small-world property is known at least since Milgram's famous experiment \cite{} and is arguably the most popular of all features of complex networks. For instance, the average distance of the Facebook graph was recently established to be just $4.74$\cite{milgram1967small}. \s
\subsection*{Definitions and conventions}
From now on, we consider directed graphs defined by a set $N$ of $n$ nodes and $A \subseteq N \times N$ of arcs. We write $x \to y$ when $(x,y)\in A$ and call $x$ and $y$ the source and the target of the arc, respectively. \s
\clearpage
\subsection*{Aim of the project}
The Aim of the project is to study the small-world phenomenon in location-based (social) networks. As test cases, we consider three real-world datasets: Brightkite, Gowalla and Foursquare. In the next sections, we will describe the datasets and the methodology we used to extract the networks from them. \s
\nd We are interest in analyzing 4 different centrality measures:
\begin{itemize}
\item Distribution of Degree
\item Clustering coefficient
\item Average Path Length
\item Betweenness Centrality
\end{itemize}
\clearpage
\section{Theoretical background on centrality measures}
Centrality is a fundamental tool in the study of social networks: the first efforts to define formally centrality indices were put forth in the late 1940s by the Group Networks Laboratory at MIT directed by Alex Bavelas \cite{closeness}; those pioneering experiments concluded that centrality was related to group efficiency in problem-solving, and agreed with the subjects' perception of leadership. In the following decades, various measures of centrality were employed in a multitude of contexts. \s
\subsection*{Geometric measures}
We call geometric those measures assuming that importance is a function of distances; more precisely, a geometric centrality depends only on how many nodes exist at every distance. These are some of the oldest measures defined in the literature.
\paragraph*{In-degree centrality} Indegree, the number of incoming arcs $d^-(x)$, can be considered a geometric measure: it is simply the number of nodes at distance one\footnote{Most centrality measures proposed in the literature were actually described only for undirected, connected graphs. Since the study of web graphs and online social networks has posed the problem of extending centrality concepts to networks that are directed, and possibly not strongly connected, in the rest of this paper we consider measures depending on the incoming arcs of a node (e.g., incoming paths, left dominant eigenvectors, distances from all nodes to a fixed node). If necessary, these measures can be called “negative”, as opposed to the “positive” versions obtained by considering outgoing paths, or (equivalently) by transposing the graph.} . It is probably the oldest measure of importance ever used, as it is equivalent to majority voting in elections (where $x \to y$ if $x$ voted for $y$). Indegree has a number of obvious shortcomings (e.g., it is easy to spam), but it is a good baseline. \s
\nd Other notable geometric measures that we will not explore in this project, are \emph{closeness centrality}, (which is the reciprocal of the sum of distances to all other nodes, and betweenness centrality, which is the number of shortest paths that pass through a node), \emph{Lin's index} (which is the sum of the distances to all other nodes), and \emph{Harmonic Centrality} (which is a generalization of the closeness centrality). \s
author={Yang, Dingqi and Zhang, Daqing and Zheng, Vincent. W. and Yu, Zhiyong},
journal={IEEE Transactions on Systems, Man, and Cybernetics: Systems},
title={Modeling User Activity Preference by Leveraging User Spatial Temporal Characteristics in LBSNs},
year={2015},
volume={45},
number={1},
pages={129--142},
ISSN={2168-2216},
publisher={IEEE}
}
@article{closeness,
ISSN = {00932914},
URL = {http://www.jstor.org/stable/44135428},
author = {Alex Bavelas},
journal = {Applied Anthropology},
number = {3},
pages = {16--30},
publisher = {Society for Applied Anthropology},
title = {A MATHEMATICAL MODEL FOR GROUP STRUCTURES},
urldate = {2022-12-07},
volume = {7},
year = {1948}
}
@book{cohen_havlin_2010,place={Cambridge},title={Complex Networks: Structure, Robustness and Function},DOI={10.1017/CBO9780511780356},publisher={Cambridge University Press},author={Cohen, Reuven and Havlin, Shlomo},year={2010}}
@misc{https://doi.org/10.48550/arxiv.1111.4570,
doi = {10.48550/ARXIV.1111.4570},
url = {https://arxiv.org/abs/1111.4570},
author = {Backstrom, Lars and Boldi, Paolo and Rosa, Marco and Ugander, Johan and Vigna, Sebastiano},
keywords = {Social and Information Networks (cs.SI), Physics and Society (physics.soc-ph), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Physical sciences, FOS: Physical sciences},
Luca Lombardo
2 years ago