Python: Mining Twitter for GamerGate: Visualization - Revision history

Onnowpurbo at 06:35, 29 January 2017

2017-01-29T06:35:12Z

← Older revision		Revision as of 06:35, 29 January 2017
Line 7:		Line 7:
	jangan lupa instalasi Gephi.		jangan lupa instalasi Gephi.

	~~Again, we’ll assume that our tweets are collected in a text file,~~ “gamergate.txt”. ~~Let’s pull the~~ data ~~out of the~~ text file ~~into a new~~ data ~~frame.~~		Asumsinya tweet berhasil dikumpulkan di “gamergate.txt”. Script berikut akan mengambil data dari text file dan memasukan ke frame data yang baru,

	import json		import json

Onnowpurbo at 06:28, 29 January 2017

2017-01-29T06:28:35Z

@@ Line 1: / Line 1: @@
-In the previous posting, I went over how to connect to Twitter’s streaming API using a connector app and the Tweepy Python library, as well as a quick overview of how to construct a Pandas dataframe from the tweets we’ve collected.
+Disini kita akan extrak semua informasi yang dibutuhkan menggunakan NetworkX untuk membuat directed graph dan mem-visualisasikan menggunakan Gephi untuk melihat siapa me-retweet siapa, mencatat umur tweet dalam hari, jumlah follower setiap user agar dikemudian hari dapat di filter jika kita menginginkannya.
-In this posting, we’ll extract all of the information we’ll need to use NetworkX to create a directed graph that we can visualize in Gephi of who’s retweeting whom, keeping track of the age in days and the number of followers that each user has so we can filter on those factors if we like.
+Instalasi NetworkX,
   sudo pip install networkx
-and download and install Gephi.
+jangan lupa instalasi Gephi.
 Again, we’ll assume that our tweets are collected in a text file, “gamergate.txt”. Let’s pull the data out of the text file into a new data frame.
@@ Line 148: / Line 146: @@
 The last thing we did was to save out a GEFX file we can then read into Gephi. Start Gephi up, and open our file; we called ours “ggrtages.gexf”.

Onnowpurbo at 05:07, 29 January 2017

2017-01-29T05:07:18Z

Show changes

Onnowpurbo at 05:03, 29 January 2017

2017-01-29T05:03:05Z

← Older revision		Revision as of 05:03, 29 January 2017
Line 3:		Line 3:
	In this posting, we’ll extract all of the information we’ll need to use NetworkX to create a directed graph that we can visualize in Gephi of who’s retweeting whom, keeping track of the age in days and the number of followers that each user has so we can filter on those factors if we like.		In this posting, we’ll extract all of the information we’ll need to use NetworkX to create a directed graph that we can visualize in Gephi of who’s retweeting whom, keeping track of the age in days and the number of followers that each user has so we can filter on those factors if we like.

	First, if you don’t have NetworkX, install it with pip, and download and install Gephi.		First, if you don’t have NetworkX, install it with pip,

			sudo pip install networkx

			and download and install Gephi.

	Again, we’ll assume that our tweets are collected in a text file, “gamergate.txt”. Let’s pull the data out of the text file into a new data frame.		Again, we’ll assume that our tweets are collected in a text file, “gamergate.txt”. Let’s pull the data out of the text file into a new data frame.

Onnowpurbo: Created page with "In the previous posting, I went over how to connect to Twitter’s streaming API using a connector app and the Tweepy Python library, as well as a quick overview of how to con..."

2017-01-29T05:01:54Z

Created page with "In the previous posting, I went over how to connect to Twitter’s streaming API using a connector app and the Tweepy Python library, as well as a quick overview of how to con..."

New page

In the previous posting, I went over how to connect to Twitter’s streaming API using a connector app and the Tweepy Python library, as well as a quick overview of how to construct a Pandas dataframe from the tweets we’ve collected.

In this posting, we’ll extract all of the information we’ll need to use NetworkX to create a directed graph that we can visualize in Gephi of who’s retweeting whom, keeping track of the age in days and the number of followers that each user has so we can filter on those factors if we like.

First, if you don’t have NetworkX, install it with pip, and download and install Gephi.

Again, we’ll assume that our tweets are collected in a text file, “gamergate.txt”. Let’s pull the data out of the text file into a new data frame.

import json
import re
import pandas as pd
from time import gmtime, mktime, strptime

tweets_data = []
tweets_file = open(tweets_data_path, "r")
for line in tweets_file:
try:
tweet = json.loads(line)
tweets_data.append(tweet)
except:
continue
#
# Clean out limit messages, etc.
#
for tweet in tweets_data:
try:
user = tweet['user']
except:
tweets_data.remove(tweet)

for tweet in tweets_data:
try:
user = tweet['text']
except:
tweets_data.remove(tweet)

#
# See how many we wound up with
#
print len(tweets_data)

#
# Pull the data we're interested in out of the Twitter data we captured
#
rows_list = []
now = mktime(gmtime())
for tweet in tweets_data:
author = ""
rtauthor = ""
age = rtage = followers = rtfollowers = 0
#
# If it was a retweet, get both the original author and the retweeter, save the original author's
# follower count and age
#
try:
author = tweet['user']['screen_name']
rtauthor = tweet['retweeted_status']['user']['screen_name']
rtage = int(now - mktime(strptime(tweet['retweeted_status']['user']['created_at'], "%a %b %d %H:%M:%S +0000 %Y")))/(60*60*24)
rtfollowers = tweet['retweeted_status']['user']['followers_count']
except:
#
# Otherwise, just get the original author
#
try:
author = tweet['user']['screen_name']
except:
continue
#
# If this was a reply, save the screen name being replied to
#
reply_to = ""
if (tweet['in_reply_to_screen_name'] != None):
reply_to = tweet['in_reply_to_screen_name']
#
# Calculate the age, in days, of this Twitter ID
#
age = int(now - mktime(strptime(tweet['user']['created_at'], "%a %b %d %H:%M:%S +0000 %Y")))/(60*60*24)
#
# Grab this ID's follower count and the text of the tweet
#
followers = tweet['user']['followers_count']
text = tweet['text']
dict1 = {}
#
# Construct a row, add it to our list
#
dict1.update({'author': author, 'reply_to': reply_to, 'age': age, 'followers': followers, 'retweet_of': rtauthor, 'rtfollowers': rtfollowers, 'rtage': rtage, 'text': text})
rows_list.append(dict1)

#
# When we've processed all the tweets, build the DataFrame from the rows
# we've collected
#
tweets = pd.DataFrame(rows_list)

Here’s a script that will iterate through the dataframe, row by row, and construct a directed graph of who’s retweeting whom. Each directed edge represented the relationship “is retweeted by”, the higher the weight of an edge, the more person B is getting retweeted by person A. Each node represents an individual ID on Twitter, and has attributes to track the number of followers and the age of the ID in days.

import networkx as nx

#
# Create a new directed graph
#
J = nx.DiGraph()
#
# Iterate through the rows of our dataframe
#
for index, row in tweets.iterrows():
#
# Gather the data out of the row
#
this_user_id = row['author']
author = row['retweet_of']
followers = row['followers']
age = row['age']
rtfollowers = row['rtfollowers']
rtage = row['rtage']
#
# Is the sender of this tweet in our network?
#
if not this_user_id in J:
J.add_node(this_user_id, attr_dict={
'followers': row['followers'],
'age': row['age'],
})
#
# If this is a retweet, is the original author a node?
#
if author != "" and not author in J:
J.add_node(author, attr_dict={
'followers': row['rtfollowers'],
'age': row['rtage'],
})
#
# If this is a retweet, add an edge between the two nodes.
#
if author != "":
if J.has_edge(author, this_user_id):
J[author][this_user_id]['weight'] += 1
else:
J.add_weighted_edges_from([(author, this_user_id, 1.0)])

nx.write_gexf(J, 'ggrtages.gexf')

The last thing we did was to save out a GEFX file we can then read into Gephi. Start Gephi up, and open our file; we called ours “ggrtages.gexf”.

gephiScreenSnapz013

You’ll get a dialog telling you how many nodes and edges there are in the graph, whether it’s directed or not, and other information, warnings, etc. Click “OK”.

gephiScreenSnapz014

Gephi will import the GEFX file. You can now look at the information it contains by clicking on the “Data Laboratory” button at the top.

gephiScreenSnapz015

Click on the “Overview” button to start working with the network. At first, it doesn’t look like anything, since we haven’t actually run a visualization on it. Before we do, we can use some of the node attributes to color nodes a darker blue based on their age.

gephiScreenSnapz016

We can use the “Ranking” settings to color our nodes. Click on the “Select attribute” popup, and choose “age”.

gephiScreenSnapz017

You can choose difference color schemes, change the spline curve used to apply color, etc., from here as well.

gephiScreenSnapz018

Click on the “Apply” button to apply the ranking to the network. The nodes will now be colored rather than gray.

gephiScreenSnapz019

Now, we’re ready to run a visualization on our data. From the “Layout” section, let’s choose “ForceAtlas 2″—it’s fast and good at showing relationships in a network.

gephiScreenSnapz020

Press the “Run” button, and let it go for a bit. A network this size—about 10K nodes and 30K edges—settled down on my MacBook Pro within five minutes or less. When you feel it’s stabilized into something interesting, press the “Stop” button, and then click on the “Preview” button at the top.

gephiScreenSnapz022

The preview panel won’t show anything at first. Click the “Refresh” button.

gephiScreenSnapz023

Gephi will render your visualization. You can use the mouse to drag it around, and you can zoom in and out with a scroll-wheel or with the “+” and “-” buttons below.

gephiScreenSnapz024

==Referensi==

* http://www.acme-dot.com/data-mining-twitter-for-gamergate-visualization/