Python: Generating Network Graph of Twitter Follower - Revision history

Onnowpurbo: /* Step 2. Process twitter data to generate an output file of relationships between twitter accounts */

2017-01-29T05:48:13Z

Step 2. Process twitter data to generate an output file of relationships between twitter accounts

← Older revision		Revision as of 05:48, 29 January 2017
Line 331:		Line 331:
	edge_exists[key] = True		edge_exists[key] = True
	view raw		view raw

	~~twitter_network.py hosted with ❤ by GitHub~~

	==Python file: twitter_network.py==		==Python file: twitter_network.py==

Onnowpurbo: /* Step 2. Process twitter data to generate an output file of relationships between twitter accounts */

2017-01-29T05:47:50Z

Step 2. Process twitter data to generate an output file of relationships between twitter accounts

← Older revision		Revision as of 05:47, 29 January 2017
Line 280:		Line 280:
	==Step 2. Process twitter data to generate an output file of relationships between twitter accounts==		==Step 2. Process twitter data to generate an output file of relationships between twitter accounts==

	~~The script below will process the~~ data ~~collected from the~~ twitter API ~~and generate an~~ edge list. ~~That is a list of relationships between~~ twitter ~~accounts~~. ~~A weight~~ value ~~is included~~, ~~this value is the~~ total ~~number of followers for the first~~ twitter account, ~~this value is retrieved from the~~ API. ~~The weight~~ value ~~can be used later to prune the~~ network graph.		Script [https://gist.github.com/mjcreativeventures/58a037a03b63355e02a3 twitter_network.py] di bawah ini akan memproses data yang dikumpulkan oleh twitter API dan membuat edge list. List ini berisi hubungan antar twitter account. Weight value dimasukan, nilai ini merupakan jumlah total follower untuk twitter account pertama, nilai ini di ambil dari API twitter. Weight value akan digunakan nanti saat kita menggambar network graph.

	import glob		import glob

Onnowpurbo: /* Python file: get_followers.py */

2017-01-29T05:44:33Z

Python file: get_followers.py

← Older revision		Revision as of 05:44, 29 January 2017
Line 261:		Line 261:


	~~==Python file: get_followers.py==~~

	I ran this script twice first without a filter on the screen name but limiting the maximum number of following accounts to 20 then again but this time filtering for accounts starting with ‘TED’ (line 102) and allowing up to 200 following accounts to be queried. This will give a mix of TED and non-TED twitter accounts. Running the script:		I ran this script twice first without a filter on the screen name but limiting the maximum number of following accounts to 20 then again but this time filtering for accounts starting with ‘TED’ (line 102) and allowing up to 200 following accounts to be queried. This will give a mix of TED and non-TED twitter accounts. Running the script:

Onnowpurbo at 05:43, 29 January 2017

2017-01-29T05:43:20Z

← Older revision		Revision as of 05:43, 29 January 2017
Line 7:		Line 7:
	sudo pip install tweepy		sudo pip install tweepy
	mkdir following		mkdir following
	mkdir ~~twitter_user~~		mkdir twitter_users

	Login ke https://dev.twitter.com/apps, dapatkan:		Login ke https://dev.twitter.com/apps, dapatkan:

Onnowpurbo at 05:42, 29 January 2017

2017-01-29T05:42:37Z

← Older revision		Revision as of 05:42, 29 January 2017
Line 1:		Line 1:
	~~Generating a network graph of Twitter followers using Python and NetworkX~~		==Persiapan==

			Instalasi

	~~Setting up tweepy, a python wrapper for the twitter api~~

	~~For collecting tweets we will use the tweepy library which is a wrapper around the twitter api. To install:~~

	sudo apt install python-pip		sudo apt install python-pip
	pip install --upgrade pip		sudo pip install --upgrade pip
	pip install tweepy		sudo pip install tweepy
			mkdir following
			mkdir twitter_user

	~~Before you can access the~~ twitter streaming api you need to register your client application (the python script) with Twitter. There is a tweepy authentication tutorial that explains step by step how to set up your app for oauth authentication.		Login ke https://dev.twitter.com/apps, dapatkan:

	~~Once you have registered with Twitter go to My Applications page on Twitter and you should see your newly created application. Click on the link and you should see the application details page:~~		* CONSUMER_KEY
			* CONSUMER_SECRET
			* ACCESS_TOKEN
			* ACCESS_TOKEN_SECRET

			==Langkah Secara umum==

	~~In this article I show you how by starting at a single twitter account we can build up a network graph of twitter followers and then visualize that network using the NetworkX library.~~

	~~The steps are:~~

	* From initial seed account collect followers using the Snowball Sampling technique.		* From initial seed account collect followers using the Snowball Sampling technique.
	* Process the collected twitter data to generate an output file of relationships between twitter accounts.		* Process the collected twitter data to generate an output file of relationships between twitter accounts.
	* Visualize network data in a network graph using the NetworkX library.		* Visualize network data in a network graph using the NetworkX library.


	==Cara Cepat==		==Cara Cepat==
Line 35:		Line 31:
	==Step 1. Collect follower data from the Twitter API==		==Step 1. Collect follower data from the Twitter API==

	~~You will need to have API keys to be able to query the Twitter API. I have written in previous articles how to do this, e.g. Collecting tweets using Python.~~		follower data akan di simpan di folder following, sesudah proses di jalankan akan data di simpan dalam CSV format.

	When you interact with the Twitter API you will learn quickly that you need to cache data as you go along. This is because the API is rate limited and you will find any script you write will halt frequently when hitting a rate limit if you don’t cache responses. The solution is to check for cached data ~~before making an API call~~, ~~if you get a cache miss then query the API and write the returned data to disk.~~

	~~I use two directories for cached~~ data~~. The directory ‘following’ contains a~~ CSV file for each twitter account queried. The name of each file is the screen name of the twitter account and the content is a tab delimited list, each row contains the twitter id, screen name and account name of a follower, up to a maximum of 200 followers.

	$ ls following/		$ ls following/
Line 58:		Line 50:
	XXXXXXXX maroo_king Marou		XXXXXXXX maroo_king Marou

	~~The second directory is called~~ ‘twitter-users’, ~~it is a cache of~~ twitter user ~~details, each file contains cached data for a twitter user including friend and follower counts and a list of follower IDs (up to a maximum of 5000 follower IDs can be queried from the API)~~.		Directory yang kedua adalah ‘twitter-users’, menyimpan twitter user detail dalam format JSON.


	$ ls twitter-users/		$ ls twitter-users/
Line 85:		Line 78:
	}		}

	~~Here is the script to collect this~~ data:		Script [https://gist.github.com/mjcreativeventures/41de04c6bbe47ee14411 get_followers.py] untuk mengumpulkan data adalah sebagai berikut,

	import tweepy		import tweepy
Line 264:		Line 257:
	print 'Sorry, could not find twitter user with screen name: %s' % twitter_screenname		print 'Sorry, could not find twitter user with screen name: %s' % twitter_screenname
	view raw		view raw
	~~get_followers.py hosted with ❤ by GitHub~~



	==Python file: get_followers.py==		==Python file: get_followers.py==

Onnowpurbo at 04:18, 29 January 2017

2017-01-29T04:18:22Z

← Older revision		Revision as of 04:18, 29 January 2017
Line 28:		Line 28:
	==Cara Cepat==		==Cara Cepat==

			python get_followers.py -s TEDxSingapore -d 3
			python twitter_network.py
			python visualize.py

Onnowpurbo at 04:06, 29 January 2017

2017-01-29T04:06:03Z

← Older revision		Revision as of 04:06, 29 January 2017
Line 24:		Line 24:
	* Process the collected twitter data to generate an output file of relationships between twitter accounts.		* Process the collected twitter data to generate an output file of relationships between twitter accounts.
	* Visualize network data in a network graph using the NetworkX library.		* Visualize network data in a network graph using the NetworkX library.


			==Cara Cepat==



	==Step 1. Collect follower data from the Twitter API==		==Step 1. Collect follower data from the Twitter API==

Onnowpurbo at 03:47, 29 January 2017

2017-01-29T03:47:16Z

← Older revision		Revision as of 03:47, 29 January 2017
Line 25:		Line 25:
	* Visualize network data in a network graph using the NetworkX library.		* Visualize network data in a network graph using the NetworkX library.

	Step 1. Collect follower data from the Twitter API		==Step 1. Collect follower data from the Twitter API==

	You will need to have API keys to be able to query the Twitter API. I have written in previous articles how to do this, e.g. Collecting tweets using Python.		You will need to have API keys to be able to query the Twitter API. I have written in previous articles how to do this, e.g. Collecting tweets using Python.
Line 56:		Line 56:
	-rw-r--r-- 1 mark mark 57K Jul 24 16:46 XXXXXXXX.json		-rw-r--r-- 1 mark mark 57K Jul 24 16:46 XXXXXXXX.json
	-rw-r--r-- 1 mark mark 6.3K Jul 24 17:01 XXXXXXXXXX.json		-rw-r--r-- 1 mark mark 6.3K Jul 24 17:01 XXXXXXXXXX.json

	... Lots more ...		... Lots more ...


	$ cat twitter-users/XXXXXXXX		$ cat twitter-users/XXXXXXXX
	{
	"name": "TEDxSingapore",		{
	"friends_count": 147,		"name": "TEDxSingapore",
	"followers_count": 12814,		"friends_count": 147,
	"followers_ids": [		"followers_count": 12814,
	XXXXXXXXXX,		"followers_ids": [
	XXXXXXXXXX,		XXXXXXXXXX,
	XXXXXXXXX,		XXXXXXXXXX,
	...		XXXXXXXXX,
	XXXXXXXXXX,		...
	XXXXXXXXXX		XXXXXXXXXX,
	],		XXXXXXXXXX
	"id": XXXXXXXX,		],
	"screen_name": "TEDxSingapore"		"id": XXXXXXXX,
	}		"screen_name": "TEDxSingapore"
			}

	Here is the script to collect this data:		Here is the script to collect this data:
Line 258:		Line 258:
	get_followers.py hosted with ❤ by GitHub		get_followers.py hosted with ❤ by GitHub

	Python file: get_followers.py		==Python file: get_followers.py==

	I ran this script twice first without a filter on the screen name but limiting the maximum number of following accounts to 20 then again but this time filtering for accounts starting with ‘TED’ (line 102) and allowing up to 200 following accounts to be queried. This will give a mix of TED and non-TED twitter accounts. Running the script:		I ran this script twice first without a filter on the screen name but limiting the maximum number of following accounts to 20 then again but this time filtering for accounts starting with ‘TED’ (line 102) and allowing up to 200 following accounts to be queried. This will give a mix of TED and non-TED twitter accounts. Running the script:
Line 277:		Line 277:
	... Lots more output ...		... Lots more output ...

	Step 2. Process twitter data to generate an output file of relationships between twitter accounts		==Step 2. Process twitter data to generate an output file of relationships between twitter accounts==

	The script below will process the data collected from the twitter API and generate an edge list. That is a list of relationships between twitter accounts. A weight value is included, this value is the total number of followers for the first twitter account, this value is retrieved from the API. The weight value can be used later to prune the network graph.		The script below will process the data collected from the twitter API and generate an edge list. That is a list of relationships between twitter accounts. A weight value is included, this value is the total number of followers for the first twitter account, this value is retrieved from the API. The weight value can be used later to prune the network graph.
Line 333:		Line 333:
	twitter_network.py hosted with ❤ by GitHub		twitter_network.py hosted with ❤ by GitHub

	Python file: twitter_network.py		==Python file: twitter_network.py==

	The output generated from this script:		The output generated from this script:
Line 351:		Line 351:
	...		...

	Step 3. Visualizing the Network using the NetworkX library		==Step 3. Visualizing the Network using the NetworkX library==

	We now have all the data we need to generate a network graph. Here are the steps used to visualize the network graph:		We now have all the data we need to generate a network graph. Here are the steps used to visualize the network graph:
Line 452:		Line 452:
	visualize.py hosted with ❤ by GitHub		visualize.py hosted with ❤ by GitHub

	Python file: visualize.py		==Python file: visualize.py==

	* Line 7 Load edge data from disk		* Line 7 Load edge data from disk

Onnowpurbo at 07:02, 22 January 2017

2017-01-22T07:02:27Z

← Older revision		Revision as of 07:02, 22 January 2017
Line 261:		Line 261:

	I ran this script twice first without a filter on the screen name but limiting the maximum number of following accounts to 20 then again but this time filtering for accounts starting with ‘TED’ (line 102) and allowing up to 200 following accounts to be queried. This will give a mix of TED and non-TED twitter accounts. Running the script:		I ran this script twice first without a filter on the screen name but limiting the maximum number of following accounts to 20 then again but this time filtering for accounts starting with ‘TED’ (line 102) and allowing up to 200 following accounts to be queried. This will give a mix of TED and non-TED twitter accounts. Running the script:

			$ mkdir following
			$ mkdir twitter_user

	$ python get_followers.py -s TEDxSingapore -d 3		$ python get_followers.py -s TEDxSingapore -d 3

Onnowpurbo at 06:57, 22 January 2017

2017-01-22T06:57:45Z

← Older revision		Revision as of 06:57, 22 January 2017
Line 8:		Line 8:

	sudo apt install python-pip		sudo apt install python-pip
			pip install --upgrade pip
	pip install tweepy		pip install tweepy