plotting.Rmd

---
title: "Social Data Science - Twitter community analysis"
author: "Dr. David Garcia"
output: rmarkdown::github_document
---

In this exercise we will analyze the community structure of the Twitter user network we retrieved in the last exercise. We will also run more advanced analyses and visualizations.

You can find this script on Rstudio Cloud: <https://rstudio.cloud/project/917383>

## Tasks:

1. Load data from large retweet network

2. Community detection and visualization

3. Permutation tests

4. Visualizing betweenness

5. K-core decomposition

# 1. Load data from large retweet network

If you can, run overnight the previous exercise (Session 08) but retrieving the full timeline of up to 3200 tweets of each user. If you could not you can still use the shorter timelines that you got or load the file "fullTimelines.RData" where you have an example for US congress members. Once you have loaded that file, we filter such that edges are retweets within the group of users in our dataset:
```{r}
library(igraph)
library(dplyr)
library(networkD3)
```
```{r}
#load("fullTimelines.RData")
#load("users.RData")
#timelines %>% filter(retweet_user_id %in% users$user_id) -> seledges
df <- read.csv('edges.csv', stringsAsFactors = F, colClasses = "character")
df <- select(df, c('from', 'to', 'weight')) %>% unique.data.frame()
df$weight <- as.numeric(df$weight)


csvvertices <- read.csv('vertices.csv', stringsAsFactors = F, colClasses = "character")
vdf <- csvvertices %>% select('id','name') %>% unique.data.frame()
vdf <- filter(vdf, vdf$name != 0)
length(df$from)

graph <- graph_from_data_frame(df, directed = F)  #, vertices = setNames(vdf, c("id","label")))
all_shortest_paths(graph, from = '4611686018451177627', to = '4611686018471180200', mode = c('all'))
```
Some analysis

```{r}
df %>% filter(!(to %in% vdf$id)) %>% select(to) %>% distinct()
```

Calculating friends and their connections
```{r}
friends <- df %>% filter(from == '4611686018467544385' | to == '4611686018467544385') %>% filter(weight > 3)
friendlist <- c(friends$to, friends$from) %>% unique()

friends <- df %>% filter(from %in% friendlist & to %in% friendlist)%>%
                  filter(weight > 3) %>% 
                  bind_rows(friends)

friendsgraph <- graph_from_data_frame(friends, directed = F)
friendscluster <- cluster_louvain(friendsgraph)
head(friends)

#df %>% filter(to == '4611686018451177627' & from == '4611686018468695677')
```



Friends of Friends
```{r}
friendsT2 <- df %>% filter(from %in% friendlist | to %in% friendlist) %>%  filter(weight > 5)
friendT2list <- c(friendsT2$to, friendsT2$from) %>% unique()

friendsT2 <- df %>% filter(from %in% friendT2list & to %in% friendT2list)%>%
                  filter(weight > 5) %>% 
                  bind_rows(friendsT2)

friendsT2graph <- graph_from_data_frame(friendsT2, directed = F)

#degrees <- degree(friendsT2graph, mode='total')
#deletedegrees <- degrees[degrees < 3]
friendsT2graph <- delete_vertices(friendsT2graph, V(friendsT2graph)[degree(friendsT2graph) < 3])


friendsT2cluster <- cluster_louvain(friendsT2graph)
modularity(friendsT2cluster)

#friendsT2 %>% filter(from == '4611686018467643119' | to == '4611686018467643119') %>% 
#              filter(from == '4611686018468695677' | to == '4611686018468695677')
```



Time to plot! Use the examples of plotting in previous exercises and visualize your network:
```{r}
#Your code here
#plot(fofgraph, vertex.size=5, vertex.label.cex = 0.4, layout = layout_nicely, edge.curved = 0.1, edge.width = 1, edge.arrow.size=0.1)
```

Repeat the plotting of the network but set the vertex.color to be the communities assigned to each node. For this you can use the membership() function of igraph. Does it look like it has a community structure?
```{r}
#plot(fofgraph, vertex.size=5, vertex.label.cex = 0.3, layout = layout_nicely, edge.curved = 0.1, edge.width = 1, edge.arrow.size=0.3, vertex.color=friendsT2cluster$membership)
```

```{r}
hist(table(membership(friendsT2cluster)))
```
The following code is an example of how to make a JavaScript dynamic visualization of the network. Can you see communities better here? Do you see nodes of the same color concentrated in parts of the figure?

remove single degree
``` {r}
# degrees <- degree(fofgraph, mode='total')
# degrees <- degrees[degrees < 3]
# smol <- delete_vertices(fofgraph, degrees)
# smollouvain <- cluster_louvain(smol)
#smol

#degrees[degrees < 3]

# graph2 <- igraph_to_networkD3(smol)
# graph2$nodes$comm <- as.character(membership(smollouvain))
# graph2$nodes$label <- vdf$name[match(graph2$nodes$name, vdf$id)]
# graph2$links$weight <- as.numeric(E(smol)$weight)
# #chargevec = -30 / (friendsoffriends$weight/sqrt(sum(friendsoffriends$weight^2))) -22
# forceNetwork(Links=graph2$links, Nodes=graph2$nodes, NodeID="label", Group="comm", zoom=T, opacity = 1, opacityNoHover = 1, fontSize = 15)

```

Friendsgraph
```{r}
graph2 <- igraph_to_networkD3(friendsgraph)
graph2$nodes$comm <- as.character(membership(friendscluster))
graph2$nodes$label <- vdf$name[match(graph2$nodes$name, vdf$id)]

forceNetwork(Links=graph2$links, Nodes=graph2$nodes, NodeID="label", Group="comm", zoom=T, opacity = 1, opacityNoHover = 1, fontSize = 15, linkDistance = JS("function(d){return 300- d['NA..3']}"), linkWidth = JS("function(d){return Math.sqrt(d['NA..3'])/3}"))
```

Friends of Friends
```{r}
library(networkD3)

graph2 <- igraph_to_networkD3(friendsT2graph)
graph2$nodes$comm <- as.character(membership(friendsT2cluster))
graph2$nodes$label <- vdf$name[match(graph2$nodes$name, vdf$id)]

forceNetwork(Links=graph2$links, Nodes=graph2$nodes, NodeID="label", Group="comm", zoom=T, opacity = 1, opacityNoHover = 1, fontSize = 12, linkDistance = JS("function(d){return 200- d['NA..3']}"), linkWidth = JS("function(d){return d['NA..3']/50}"))
```


# 3. Permutation tests

Let's compare with a random network that has the same degrees as the original one. This code generate a network that keeps node degrees but switches connections until the network is completely shuffled:
```{r}
rndnet <- sample_degseq(out.deg=degree(friendsT2graph))
```

Your turn, run the Louvain algorithm over that and calculate its modularity. Is it high?
```{r}
sample_louvain <- cluster_louvain(rndnet)
modularity(sample_louvain)
```

Now run a permutation test, repeating the above process 1000 times and saving each modularity value in a vector. Plot the histogram and the original value.
```{r}
N <- 1000
rndmods <- rep(NA,N)
for (i in seq(1,N))
{
  rndnet <- sample_degseq(out.deg=degree(friendsT2graph))
  sample_louvain <- cluster_louvain(rndnet)
  rndmods[i] <- modularity(sample_louvain)
}

hist(rndmods)
friendsT2cluster = cluster_louvain(as.undirected(friendsT2graph))
observed <- modularity(friendsT2cluster)
observed
#Your code here
```

Now calculate the p-value of that permutation test. How likely is to observe that extreme modularity under the null model?
```{r}
#Your code here
#pval = pnorm(rndmods)
```

# 4. Visualizing betweenness

Calculate the betweenness of each of the nodes in your network and plot its distribution. Is it skewed?
```{r}
betw <- betweenness(friendsT2graph)
hist(betw)
```

Since the distribution is very skewed, we calculate the logarithm of betweenness values (after adding 1 to avoid zeroes). We normalize by the maximum value to make the numbers between 0 and 1. Then we plot a network with vertex colors using a scale proportional to that value. Can you see the nodes with the highest betweeness? Do they look like the ones that have the most shortest paths passing over them?
```{r}
logbetws <- log(betw+1)/max(log(betw+1))
plot(friendsT2graph, vertex.color = gray(1-logbetws), vertex.label.cex=0.001, layout=layout_nicely, 
     vertex.size=5, edge.curved=0.1, edge.width=1, edge.arrow.width=0.1)
```

```{r}
graph3 <- igraph_to_networkD3(friendsT2graph, group = logbetws)
forceNetwork(Links=graph3$links, Nodes=graph3$nodes, NodeID="name", Group="group", zoom=T, opacity = 1, opacityNoHover = 1)
```
# 5. K-core decomposition

Now let's calculate the k-core decomposition, getting the coreness value of each node. Plot its histogram. Does it look more or less skewed than the betweenness?
```{r}
#Your code here
corns = coreness(friendsT2graph)
plot(friendsT2graph, vertex.color = corns, vertex.label.cex = 0.1, edge.arrow.size = 0.1, vertex.size = 5, edge.width = 1, edge.curved = 0.1)
```

And now we plot with colors according to the coreness of each node. Do you feel any difference to the betweenness?
```{r}
plot(friendsT2graph, vertex.color = corns, vertex.label.cex=0.05, layout=layout_nicely, 
     vertex.size=5, edge.curved=0.1, edge.width=1)
```