PPPeople PPPowered

Progress February 2011

2011-02-07T08:15:00.000-08:00

Over the last few months I've tried a number of novel approaches to try and make sense of the data that the crawler and semantifier return.

Many of these approaches (based on the Neo4J work) still have some potential ( I think ), but I wasn't able to prove it.

Recently, if only in an effort to see what's there, I went back to basics and do the simplest thing possible (almost). The Home Page now looks like this (see below)... a simple tag cloud showing the most frequent concepts and images of the people who's pages were found to contain squarish, largeish images. There is also a type-ahead scrolling search box. Showing people what is in the database, before they search is a really useful means of helping people not to search for things that aren't there.

I decided to attempt to return only related people and then overlapping concepts (for any given concept). This would mean that a search for plankton might return ...

Plankton -> (Peter Daines, Andy Bird, Rob Raiswell, Michael Krom etc) which would then subsequently return the connected concepts of (Fisheries, Aquatic ecology, Climate change, Environment etc).

The result of this really simple approach looks like this...

...and bizarrely, because the concepts are interlinked, the visualisation sorts itself out. That is to say, previously I was attempting to fingerprint the connections between concepts and then sort them, but found that the visualisation ( trying to find the best layout for interconnected concepts) actually does the same job, but in 2D space rather than in a matrix.

A rough beta of this work is online here http://pppeople.collabtools.org.uk/ . Be gentle with it, it's not finished. I now want to see if I can create some embed code that might be usefully added to another site - showing a mini collection of related concepts and URLs.

Neo4J + Python Crawler + Open Calais + Gephi

2010-08-06T09:24:00.000-07:00

So, today after having spent the morning getting our LDAP server to talk to our Cyn.in instance (it worked!) I thought I'd have a bash at this. The plan was/is to create a crawler that gets a web page, finds the pages that it links to (including Word or PDF files), pumps any data found at the Open Calais API and saves the semantic Entities returned into a Neo4J database then go and look at it with the Gephi visualising tool ( see previous posts ).

This is all kinda new to me, I have no idea if what I'm doing makes sense, but I hope that once I can "look" at data, I'll be able to figure out a way of pruning it into something usable.

The (ropey) code is here and the visualisation is shown above. I have no idea if this will be "traversable" yet, but it kind of proves to me that it's doable. Ideally I want to crawl a given pile of pages into one big soup and then get from page B to page X and "discover" the shortest route between them...

How hard can it be :-)


5    import urllib2, urllib,  traceback, re, urlparse, socket, sys, random, time 
6    from pprint import pprint 
7    import codecs, sys 
8    from calais import Calais      #http://code.google.com/p/python-calais/ 
9    from scraper import fetch     #http://zesty.ca/python/scrape.py 
10    
11   streamWriter = codecs.lookup('utf-8')[-1] 
12   sys.stdout = streamWriter(sys.stdout) 
13    
14   API_KEY = 'YOUR_KEY_HERE' 
15   calaisapi = Calais(API_KEY, submitter="python-calais demo") 
16    
17   def analyze(url=None): 
18       result = calaisapi.analyze_url( url ) 
19       return result 
20    
21   def neo_result(result, url): 
22       import neo4j 
23    
24       db = neo4j.GraphDatabase( "simple_neo_calais_test" ) 
25    
26       result.print_summary( ) 
27        
28       with db.transaction: 
29           # Create the page index 
30           pages = db.index("Pages", create=True) 
31           page_node = pages[url] # does this page exist yet? 
32           if not page_node: 
33               page_node = db.node(url=url) # create a page 
34               pages[ url ] = page_node # Add to index 
35               print "Created:" , url 
36           else: 
37               print "Exists already:" , url 
38    
39           print len(result.entities), "Calais Entities" 
40           for e in result.entities: 
41               print result.doc['info']['externalID']    #URL 
42               entity_type =  e['_type'] 
43               entity_value = e['name'] 
44               relevance = e['relevance'] 
45               instances = e['instances'] 
46    
47               print entity_type, entity_value, relevance, instances #instances is a list of contexts 
48    
49               #Create an entity 
50               entity = db.node(value=entity_value, relevance=relevance ) 
51               entity_type = db.node( name= entity_type ) 
52               entity_type.is_a( entity )    # e.g Amazon is_a Company 
53               page_node.has( entity_type ) 
54    
55       db.shutdown() 
56     
59    
60   def print_result(result): 
61       'Custom code to just show certain bits of the result obj' 
62       result.print_summary( ) 
63    
64       print "Entities" 
65       for e in result.entities: 
66           print e['_type'], ":", (e['name']) 
67           print e.keys() 
68           print 
69    
70       print "Topics" 
71       for t in result.topics: 
72           print t['category'] 
73           #print t 
74    
75       print 
76       print "Relations" 
77       print result.print_relations( ) 
78    
79    
80    
81   suffixes_to_avoid=[ 'css','js','zip', ] 
82   def get_links( data, url='',suffixes_to_avoid=suffixes_to_avoid ): 
83       'I know I should use BeautifulSoup or lxml' 
84       links = [ ] 
85       icos = [ ] 
86       feeds = [ ] 
87       images = [ ] 
88       try: 
89           found_links = re.findall(' href="?([^\s^"\'#]+)', data)# think this strips off anchors 
90    
91           for link in found_links: 
92               # fix up relative links 
93               link = urlparse.urljoin(url, link) 
94               link = link.replace("/..", "") # fix relative links 
95    
96               #check to see if path is just "/"... for example, http://theotherblog.com/ 
97               path = urlparse.urlsplit(link)[2] 
98               if path  == "/": 
99                   link = link[:-1]#take off the trailing slash (or should we put it on?) 
100   
101              #just in case fixups 
102              link = link.replace("'", "") 
103   
104              if link not in links and link[:7] == 'http://' : #avoid mailto:, https:// 
105                  if "." in path: 
106                      suffix_found = path.split(".")[-1] 
107                      print suffix_found, link 
108                      if suffix_found  in suffixes_to_avoid: 
109                          pass 
110                      else: 
111                          if suffix_found == 'ico': 
112                              icos.append( link ) 
113                          elif suffix_found == 'rss' or suffix_found=='xml': 
114                              feeds.append ( link ) 
115                          elif suffix_found in ['gif', 'png', 'jpg', 'jpeg']: 
116                              images.append( link ) 
117                          else: 
118                              links.append( link ) 
119                  else: 
120                      links.append( link ) 
121   
122   
123      except Exception, err: 
124          print err 
125   
126      return links, icos, images, feeds 
127   
128  def get_images(content, url): 
129      #ico 
130      #feeds 
131      #images 
132      #text 
133      found_images = re.findall(' src="?([^\s^"\'#]+)', content) 
134      images = [ ] 
135   
136   
137      for image in found_images: 
138          image = urlparse.urljoin(url, image) 
139          image = image.replace("/..", "") 
140          path = urlparse.urlsplit(image)[2] 
141           
142          if path[-3:] == ".js": 
143              pass 
144          else: 
145               images.append( image ) 
146      return images 
147   
148  def get_icos(content, url): 
149      #href="/static/img/favicon.ico" 
150      #image/x-icon  
151      found_images = re.findall(' href="?([^\s^"\'#]+\.ico)', content) 
152      images = [ ] 
153   
154   
155      for image in found_images: 
156          image = urlparse.urljoin(url, image) 
157          image = image.replace("/..", "") 
158          path = urlparse.urlsplit(image)[2] 
159   
160          if path[-3:] == ".js": 
161              pass 
162          else: 
163               images.append( image ) 
164               
165      print images 
166      if len(images)> 0: 
167          print len(images), "icos" 
168          image = images[ 0 ] #just get the first 
169      else: 
170          #try a default... you never know... might be aliased anyway... 
171          scheme, domain, path, query, x = urlparse.urlsplit( url ) 
172          image = '%s://%s/favicon.ico' % (scheme, domain) 
173           
174      return image 
175   
176   
177  mimes_u_like = [ 'text/html', 'application/pdf', ] 
178  def get( url, mimes_u_like=mimes_u_like, fetch_links=False ): 
179      url, status, message, headers, content  = fetch(url) 
180      mime = headers['content-type'].split(";")[0].strip() 
181   
182      page = {'url':url, 
183              'mime':mime, 
184              'status':status, 
185              'message':message, 
186              'headers':headers, 
187              'content':content, 
188              'links':[], 
189              'images':[], 
190              'feeds': [] } 
191   
192   
193      ico =  get_icos( content, url ) 
194      page['ico'] = ico 
195       
196      images_found =  get_images( content, url ) 
197      page['images'] = images_found 
198       
199      if fetch_links == False: 
200          return page     # this is wrong... I want to discover the the links without necessarily getting their data 
201      else: 
202          links_found, icos, images, feeds = get_links( content, url ) 
203          print url 
204          print len( links_found ), "links found" 
205          links = [ ] 
206          print len( icos ), "icos" 
207          print len( images ), "images" 
208          print images 
209          print len( feeds ), "feeds" 
210   
211   
212          for link in links_found: 
213              url, status, message, headers, content  = fetch(link) 
214              mime = headers['content-type'].split(";")[0].strip() 
215              if mime in mimes_u_like: 
216                  print "\t", status, mime, link 
217                  if status == 200: 
218                      links.append( {'url':url, 
219                                     'status':status, 
220                                     'message':message, 
221                                     'headers':headers, 
222                                     'mime':mime, 
223                                     'content':content} 
224                                    ) 
225   
226          print len(links) , "Used" 
227          page['links'] = links 
228          return page 
229   
230  if __name__ == '__main__': 
231      url = 'http://theotherblog.com' 
232      print "Getting:", url 
233      page = get( url, fetch_links=True ) 
234      
235      result = analyze( url ) 
236      neo_result ( result, url ) 
237   
238      print "Getting %s links.." % str(len(page['links'])) 
239      for link_dict in page['links']: 
240          print "Getting link:", link_dict['url'] 
241          try: 
242              result = analyze( link_dict['url'] ) 
243              neo_result ( result, link_dict['url'] ) 
244          except Exception, err: 
245              print    link_dict['url'], err 
246   
247      print "Done!"

Django and Neo4J

2010-08-04T07:51:00.000-07:00

So, today I was tempted by the approach taken in these slides about Neo4J and Django. My hope was/is that whilst the Neo4J implementation isn't complete, because I imagine many of the concepts simply won't easily map from a graph database to a SQL one, at least I might be able to work in a more familiar way in order to be able to learn about neo4j.

I started by grabbing the files from the model folder from here.... adding...

NEO4J_RESOURCE_URI = '/Users/tomsmith/neo4django_db'

...to my settings.py file and then adding this code to a models.py file...

from neo4j.model import django_model as models

class Actor(models.NodeModel):
   name= models.Property(indexed=True)
   href = property(lambda self:('/actor/%s/' % self.node.id))

   def __unicode__(self):
return self.name

class Movie(models.NodeModel):
   title = models.Property(indexed=True)
   year = models.Property()
   href = property(lambda self:('/movies/%s/' % self.node.id))

   actors = models.Relationship(Actor,type=models.Outgoing.ACTS_IN,related_name="acts_in",)

   def title_length(self):
return len(self.title)

   def __unicode__(self):
return self.title

... and then from within python manage.py shell I could...

>> import stuff.models as m

>> movie = m.Movie(title="Fear and Loathing in Las Vegas",year=1998)
>> movie.save()

>>> movie.save()
>>> movie.id
4L

>>>>> movie = m.Movie(title="The Brothers Grimm",year=2005)

>>> movie.save()

>>> movie.id

>>> movie = m.Movie(title="Twelve Monkeys",year=1995)

>>> movie.save()

>>> movie = m.Movie(title="Memento",year=2000)

>>> movie.save()

... now I can...

>> movies = m.Movie.objects.all()

>>> for movie in movies:print movie.id, movie, movie.href

...

7 Memento /movies/7/

6 Twelve Monkeys /movies/6/

5 The Brothers Grimm /movies/5/

4 Fear and Loathing in Las Vegas /movies/4/

3 The Book of Eli /movies/3/

1 10,000 years B.C. /movies/1/

>>>

... which is fantastic! Don't you think?! The django_model is handling the db.transaction stuff. I can even do a ...

>> reload( m )

... and it doesn't break (because the database is already connected) as my earlier code did. (I think I can even run my django instance AND be connected to the neo4j database at the same time.. phew!).

What I think is fantastic about it is that using this approach I get to work with objects in a familiar way. And by that, I mean, having a schema-less database is all well and good, but almost the first thing I seem to find myself doing is creating similar types of objects. So being able to create classes (maybe even with subclasses) seems perfect for my needs ... in theory. In theory, I'd like to be able to define classes with properties but then on-the-fly maybe add extra properties to objects (without changing my class definition).

AND I can add methods to my classes... or can I ? I added a simple function to my Movie class...

def title_length(self):

return len(self.title)

...and ran...

>>> movies = m.Movie.objects.all()

>>> for movie in movies:print movie.id, movie, movie.href, movie.title_len()

...and got...

AttributeError: 'Movie' object has no attribute 'title_len'

... Boo! At this point after doing looking at the Movie objects, I found ...

>>> dir( m.Movie.objects )

['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slotnames__', '__str__', '__subclasshook__', '__weakref__', '_copy_to_model', '_inherited', '_insert', '_set_creation_counter', '_update', 'aggregate', 'all', 'annotate', 'complex_filter', 'contribute_to_class', 'count', 'create', 'creation_counter', 'dates', 'defer', 'distinct', 'exclude', 'extra', 'filter', 'get', 'get_empty_query_set', 'get_or_create', 'get_query_set', 'in_bulk', 'iterator', 'latest', 'model', 'none', 'only', 'order_by', 'reverse', 'select_related', 'update', 'values', 'values_list']

...so I did a ...

>>> movies = m.Movie.objects.all().order_by('year')

...and got ...

Traceback (most recent call last):

File "", line 1, in

AttributeError: 'NodeQuerySet' object has no attribute 'order_by'

...and the same is true for .get_or_create() or .values() or many other methods.

... so (and this isn't a criticism) it seems that the Django model is a "work in progress" (the cat ate his source code) with some of the methods still to be completed. At this point I should probably start looking at the django_model source code and begin adding to the functionality it offers... except ...

I probably don't have the ability to create a database adapter, especially for one I don't understand
I don't know if shoe-horning neo4j into Django is a good idea. I'm not saying it isn't ( so far, it's tantalising ) I'm just saying I don't know.
When I find errors I'm not sure if it's me or not...

...for example, I tried...

>>> actor = m.Actor(name="Joseph Melito")

>>> actor.save()

>>> actor.id

10L

>>> actor = m.Actor(name="Bruce Willis")

>>> actor.save()

>>> actor = m.Actor(name="Jon Seda")

>>> actor.save()

...and all was fine and dandy and then I tried ...

>>> movie = m.Movie.objects.get(title="Twelve Monkeys")

>>> movie.actors.add ( actor )

... and got...

Traceback (most recent call last):

File "", line 1, in

File "/opt/local/lib/python2.5/site-packages/Neo4j.py-0.1_SNAPSHOT-py2.5.egg/neo4j/model/django_model/__init__.py", line 623, in __get__

return self._get_relationship(obj, self.__state_for(obj))

File "/opt/local/lib/python2.5/site-packages/Neo4j.py-0.1_SNAPSHOT-py2.5.egg/neo4j/model/django_model/__init__.py", line 708, in _get_relationship

states[self.name] = state = RelationshipInstance(self, obj)

File "/opt/local/lib/python2.5/site-packages/Neo4j.py-0.1_SNAPSHOT-py2.5.egg/neo4j/model/django_model/__init__.py", line 750, in __init__

self.__removed = pyneo.python.set() # contains relationships

AttributeError: 'module' object has no attribute 'set'

... I wonder if Tobias has done more work on the Relationships class at all?

Visualisation of a Neo4J database with Gephi

2010-08-03T08:12:00.000-07:00

1. Get the version of Gephi app that can read neo4j databases (not the main one)
bzr branch http://bazaar.launchpad.net/~bujacik/gephi/support-for-neo4j

2. Get Netbeans, open the project you've just downloaded and build it as a Mac application (you'll find it in a folder called dist). As it happens, Netbean is quite a lovely python IDE, once you've added a few plugins. I added a python plugin, a regular expression one and found I could pretty much use the python debugger out of the box (more than I can say for Eclipse) and watch variables etc. Very nice.

3. Run the code from yesterday and open the DB that gets created with Gephi... alter the layouts and you get something like this...

... which was a small crawl of a small site (showing IDs).

I have no absolutely no idea what this all means but isn't it lovely to look at? I think I'm onto something.

Working Neo4J / Python code

2010-08-03T02:18:00.000-07:00

So, this sort of works. The team at neo4j have made the iteration work for me... Usage:

>>python2.6 neo.py http://www.wherever.com

... and what it does it go to that web site and grabs a few links, following them, adding the pages and their data to a neo4j graph database.

import neo4j #See http://components.neo4j.org/neo4j.py/
import urllib2, traceback, re, urlparse, socket, sys, random, time

socket.setdefaulttimeout(4) #seconds

'''This is my attempt to begin to make proper classes and functions to talk to the neo4j database, it's not meant to be fancy or clever, I just want to be able to CRUD ok-ish'''


try:
 db = neo4j.GraphDatabase("crawler_example3_db")
 with db.transaction:
  #How do we delete an index... which might have old nodes in? This doesn't work...any ideas... how do I empty an index, should I? Does deleting a node delete its refererence from the index
  '''try:
   for ref in db.index('pages'):
    ref.delete()
  except Exception, err:
   print err
  '''
  pages = db.index("pages", create=True) # create an index called 'pages'
  #print "Index created"
 #return db, pages
except Exception, err:
 print err
 #print "db:", db


#############     UTILITY FUNCTIONS         #############

def get_links( data, url=''):
 'I know I should use BeautifulSoup or lxml, but for simplicity it's a regex..ha'
 links = [ ]
 try:
  found_links = re.findall(' href="?([^\s^"\'#]+)', data)
  
  for link in found_links:
   #print link
   link = urlparse.urljoin(url, link)
   link = link.replace("/..", "") #fix relative links
   link = link.replace("'", "")
   if link not in links and link[:7] == 'http://' :#avoid mailtos
    links.append( link )
 except Exception, err:
  print err
  
 return links
 
def wget(url):
 try:
  handle = urllib2.urlopen(url)
  if 'text/html' in handle.headers['content-type']:
   data = unicode(handle.read(),  errors='ignore')
   return data
 except Exception, e:
  print "Error wgetting", e, url
  #print traceback.print_exc()
 return None

def delete_all_pages():
 with db.transaction:
  try:
   for node in db.node:
    id  = node.id
    node.delete( )
    print id, "deleted"
  except Exception, err:
   print err
   pass # fails on last iteration?
 print "All pages deleted!"


def create_page( url, code=200 , follow=True):
 '''this creates a page node then adds it to the "pages" index, if it's there already it'll get it'''
 
 # Get the actual page data from the web...
 data = wget( url ) #get the actual html
 if not data:
  return None
  print "Boo!", url
 else:
  data_len = len(data)
  page_node = pages[url] # does this page exist yet?
  if not page_node:
   page_node = db.node(url=url, code=code, data=str(data), data_len=data_len) # create a page
   pages[ url ] = page_node # Add to index
   print "Created:" , url
  else:
   print "Exists already:" , url
   
  #Now create a page for every link in the page
  if follow == True:
   print "\tfollowing links"
   i = 0
   links = get_links(data, url)
   print len( links ) , "found"
   if len(links) > 20:
    links = links[:19]
   for link in links:
    print "\tcreating:",  link
    create_page( link, follow=False)
  return page_node

def get_one(url):
 'given a url with return a node obj'
 with db.transaction:
  try:
   node = pages [ url ]
   if node == None:
    print "Node is none! Creating..."
    create_page( url, code=200 , follow=False)
   return node
  except Exception, err:
   print err
   
 return None
 

def list_all_pages( ):
  # Just iterate through the pages to make sure the data in in there...
  print "Listing all pages..."
 with db.transaction: 
  for node in db.node:
   try:
    print node
    #print node['data'][:150], "..."
    print 
   except Exception, err:
    print err
 print "...done listing!"


def delete_one( url ):
 ''
 with db.transaction:
  try:
   node = pages[ url ]
   if node:
    node.delete( )
    print "Node with id=", node.id, "deleted"
    #delete from index
    del pages[url]
  except Exception, err:
   print "Node probably not found:", err
   print dir(node) # let's have a look



def find_links_between_pages( ):
 #'Just iterate through the pages to make sure the data in in there...'
  print "Linking all pages..."
 with db.transaction:
  try: 
   for node in db.node:
    try:
     print str(node), node['url'], node['data_len']
     links = get_links( node['data'], node['url'])
     for link in links:
      try:
       print link
       #look to see if a node with that url exists, if it doesn't it's created...
       other_node = get_one( link )
       if not other_node:
        pass
       else:
        other_node.links_to( node )
      except Exception, err:
       print err
     print
    except Exception, err:
     print err
    print 
  except Exception, err:
   pass # fails on last iteration?
   print err


class Backlink(neo4j.Traversal):
 types = [neo4j.Outgoing.links_to]
 order = neo4j.DEPTH_FIRST
 stop = neo4j.STOP_AT_END_OF_GRAPH
  
 def isReturnable(self, pos):
  if pos.is_start: return False
  else:return pos.last_relationship.type == 'links_to'


if __name__ == '__main__':

 #### LET'S GET STARTED ######

try:
  sys.argv[1]
  url = sys.argv[1]
 except:
  print "No url passed, using 'http://diveintopython.org/'"
  url = 'http://diveintopython.org/'
  
 print "Starting:", url
 
 ######## DELETE ALL THE PAGES WE HAVE SO FAR ############
 # Avoiding this because of ...
 #jpype._jexception.RuntimeExceptionPyRaisable: org.neo4j.graphdb.NotFoundException: Node[0] 
 #... errors later...
 #print "Deleting existing pages..."
 #delete_all_pages( ) #we may want to add new data to each page... forget this for now
 
 ########        ADD SOME ACTUAL PAGES        #################
 #print "Creating some pages "
 with db.transaction:
  create_page(  url, 200, follow=True ) #Also fetches some pages linked from this page
 
 ########    NOW GET SOME NODES OUT     ############### 
 print
 print "Has our data made it to the database? Let's see..."
 # Do some fishing for nodes...
 with db.transaction:
  try:
   node = get_one( url )
   print '\tget one:' + str( node ) , "... oh yes!"
   print "\tid:", node.id , "url:", node['url'], "data-length:", node['data_len']
   print 
   
   #This should fail
   #print "This SHOULD fail.."
   #node = get_one( 'http://www.pythonware.com/daily/' )
   #print 'get one:' + str( node ) 
   #print "id:", node.id , "url:", node['url'], "data length:", node['data_len']
   #print 
   
  except Exception, err:
   print "Probably not a node with that URL"
   print err 
 print
 
 #########  TRY TO ITERATE ALL PAGES FETCHED ################
 list_all_pages(  )
 
 #########    TRY TO DELETE ONE ################
 #delete_one( url ) 
 # Now let's see if it has gone
 #list_all_pages( ) #or maybe later
 
 ######### LET'S LOOK FOR RELATIONSHIPS BETWEEN PAGES ################
 print
 print "Doing linking pages..."
 find_links_between_pages( ) #goes to every page, looking for other pages in the database
 
 print
 print "shutting down, saving all the changes, etc"
 db.shutdown()

Network slides...

2010-08-02T08:56:00.001-07:00

...to read later...

A Perspective on Graph Theory and Network Science

View more presentations from Marko Rodriguez.

Python and Neo4J - Could this be the Funky Data Model?

2010-08-02T08:17:00.000-07:00

For many years now, I've been chasing the funky data model, a database where you can change your mind about the structure of the data on the fly, without losing all chance of ever getting the data out again. Neo4J promises that way of working.

So, before I get all fancy, I'm trying to make a painfully simple python crawler that will go to a URL, find a few pages linked from that URL, add them to the database and then create relationships between the pages based on their links. Simple. Ahem.

This code doesn't quite work at the moment... but it's almost there....

import neo4j #See http://components.neo4j.org/neo4j.py/
import urllib2, traceback, re, urlparse, socket, sys

socket.setdefaulttimeout(4) #seconds

'''This is my attempt to begin to make proper classes and functions to talk to the neo4j database, it's not meant to be fancy'''

def setup():
        ########  DO SOME SETUP STUFF ############
        db = neo4j.GraphDatabase("crawler_example_db")
        with db.transaction:
                #How do we delete an index... which might have old nodes in? This doesn't work...any ideas... how do I empty an index, should I? Does deleting a node delete its refererence from the index
                try:
                        for ref in db.index('pages'):
                                ref.delete()
                except Exception, err:
                        print err

                pages = db.index("pages", create=True) # create an index called 'pages'
                print "Index created"
        return db, pages

#############     UTILITY FUNCTIONS         #############

def get_links( data, url=''):
        'I know I should use BeautifulSoup or lxml'
        links = [ ]
        try:
                found_links = re.findall(' href="?([^\s^"\'#]+)', data)

                for link in found_links:
                        #print link
                        link = urlparse.urljoin(url, link)
                        link = link.replace("/..", "") #fix relative links

                        link = link.replace("'", "")
                        if link not in links and link[:7] == 'http://' :#avoid mailtos
                                links.append( link )
        except Exception, err:
                print err

        return links

def wget(url):
        try:
                handle = urllib2.urlopen(url)
                if 'text/html' in handle.headers['content-type']:
                        data = unicode(handle.read(),  errors='ignore')
                        return data
        except Exception, e:
                print "Error wgetting", e, url
                #print traceback.print_exc()
        return None

def delete_all_pages():
        with db.transaction:
                try:
                        for node in db.node:
                                id  = node.id
                                node.delete( )
                                print id, "deleted"
                except Exception, err:
                        print err
                        pass # fails on last iteration?
        print "All pages deleted!"

def create_page(db, pages, url, code=200 , follow=True):
        '''this creates a page node then adds it to the "pages" index, if it's there already it'll get it'''

        # Get the actual page data from the web...
        data = wget( url ) #get the actual html
        if not data:
                return None
        else:
                data_len = len(data)
                page_node = pages[url] # does this page exist yet?
                if not page_node:
                        page_node = db.node(url=url, code=code, data=data, data_len=data_len) # create a page
                        pages[ url ] = page_node # Add to index
                        print "Created:" , url
                else:
                        print "Exists already:" , url

                #Now create a page for every link in the page
                if follow == True:
                        print "\tfollowing links"
                        i = 0
                        links = get_links(data, url)
                        print len( links ) , "found"
                        if len(links) > 20:
                                links = links[:19]
                        for link in links:
                                print "\tcreating:",  link
                                create_page(db, pages, link, follow=False)
                return page_node

def get_one(url):
        'given a url with return a node obj'
        with db.transaction:
                try:
                        node = pages [ url ]
                        if node == None:
                                print "Node is none!"
                        return node
                except Exception, err:
                        print err

        return None


def list_all_pages(db):
         # Just iterate through the pages to make sure the data in in there...
         print "Listing all pages..."
        with db.transaction:
                for node in db.node:
                        try:
                                print node
                                #print node['data'][:150], "..."
                                print
                        except Exception, err:
                                print err
        print "...done listing!"

def delete_one( url ):
        ''
        with db.transaction:
                try:
                        node = pages[ url ]
                        if node:
                                node.delete()
                                print "Node with id=", node.id, "deleted"
                except Exception, err:
                        print "Node probably not found:", err
                        print dir(node) # let's have a look

def find_links_between_pages(db):
#'Just iterate through the pages to make sure the data in in there...'
        with db.transaction:
                try:
                        print "Linking all pages..."
                        for node in db.node:
                                try:
                                        print str(node), node['url'], node['data_len']
                                        links = get_links( node['data'], node['url'])
                                        for link in links:
                                                try:
                                                        #look to see if a node with that url exists
                                                        other_node = get_one( link )
                                                        if not other_node:
                                                                pass
                                                        else:
                                                                other_node.links_to( node )
                                                except Exception, err:
                                                        print err
                                except Exception, err:
                                        print err
                                print
                except Exception, err:
                        pass # fails on last iteration?
                        print err

if __name__ == '__main__':

        #### LET'S GET STARTED ######
        db, pages = setup( ) # I can't connect on load, because then I can't reload(this_module) because it is already connected
        try:
                sys.argv[1]
                url = sys.argv[1]
        except:
                print "No url passed, using 'http://diveintopython.org/'"
                url = 'http://diveintopython.org/'

        print "Starting:", url

        ######## DELETE ALL THE PAGES WE HAVE SO FAR ############
        # Avoiding this because of ...
        #jpype._jexception.RuntimeExceptionPyRaisable: org.neo4j.graphdb.NotFoundException: Node[0]
        #... errors later...
        #print "Deleting existing pages..."
        #delete_all_pages( ) #we may want to add new data to each page... forget this for now

        ########        ADD SOME ACTUAL PAGES        #################
        #print "Creating some pages "
        with db.transaction:
                create_page( db, pages, url, 200 ) #Also fetches some pages linked from this page




        ########    NOW GET SOME NODES OUT     ###############
        print
        print "Has our data made it to the database? Let's see..."
        # Do some fishing for nodes...
        with db.transaction:
                try:
                        node = get_one( url )
                        print '\tget one:' + str( node ) , "... oh yes!"
                        print "\tid:", node.id , "url:", node['url'], "data-length:", node['data_len']
                        print

                        #This should fail
                        #print "This SHOULD fail.."
                        #node = get_one( 'http://www.pythonware.com/daily/' )
                        #print 'get one:' + str( node )
                        #print "id:", node.id , "url:", node['url'], "data length:", node['data_len']
                        #print

                except Exception, err:
                        print "Probably not a node with that URL"
                        print err
        print

        db.shutdown( )
        db, pages = setup( )

        #########  TRY TO ITERATE ALL PAGES FETCHED ################
        list_all_pages( db )

        #########    TRY TO DELETE ONE ################
        #delete_one( url )
        # Now let's see if it has gone
        #list_all_pages( ) #or maybe later

        ######### LET'S LOOK FOR RELATIONSHIPS BETWEEN PAGES ################
        print "Doing linking pages..."
        find_links_between_pages( ) #goes to every page, looking for other pages

        print "shutting down, saving all the changes, etc"
        db.shutdown()

Frustrating APIs

2010-07-13T08:40:00.000-07:00

It's been a frustrating week.

At first I wanted to work with the Twitter API. I found that Oauth is a bit of pig to work with, and potentially not what what I want. I created a HTML-scraping version of the API that might just get me the information I need simply but it wasn't long before twitter realised what was up and blocked my IP.

Then I wanted to work with the Delicious API. I found their API is a bit of pig to work with, and potentially not what what I want. I created a HTML-scraping version of the API that might just get me the information I need simply but it wasn't long before Yahoo realised what was up and blocked my IP.

Next I wanted to work with the Google Search API. I found their API is a bit of pig to work with, and potentially not what what I want. I created a HTML-scraping version of the API that might just get me the information I need simply but it wasn't long before Google realised what was up and blocked my IP.

I'm spotting a pattern. APIs are crap.

Discovery and Construction of Authors’ Profile from Linked Data

2010-04-03T06:27:00.000-07:00

This looks useful and relevant. No idea where I found it, will dig it out later.

The Next Sprint

2010-04-01T04:38:00.000-07:00

Before I tidy up the code so far (it's a mess trust me) ... the next Sprint will be all about integration. The reason for this is that it's all very well having a noodley toy with which people can browse and explore and vanity surf but unless it is integrated into a tool that someone uses very regularly then it will never have the chance to prove itself.

The tool we will be using as a collaborative platform will be Cyn.in because ...

it has a plethora of tools, wikis, blogs, discussions, events etc
it has a lovely UI
it is written in Plone and python is my tool of choice

My work following Easter will look something like this...

Find out more about Plone (I've bought a book :-)
Create a script that inserts objects into the ZODB (in this case events crawled from the University site)
Create a new "content type"... I would quite like to create a "Location" object.
Create (or install) an RSS reader so we can add tweets, feed etc.
See if there's a LATEX option for the kinky scientists
Make a portlet to display this stuff
Lighten the overall look n feel.
Make a "content parser" so that key terms in a taxonomy can be automatically replaced (e.g. SIPIG (a commitee), VC011 (my room number) and dps4 (my boss) with links to the relevant items.
Create (or install) a JSON reader so that we can potentially dynamically find "related people" and display them on peoples' profiles

That looks like two weeks work to me (or more).... I may be gone sometime...

Put It All Together... Version 0.1

2010-03-31T06:40:00.000-07:00

I couldn't resist connecting the web site tag cloud terms to people associated with those terms to a person's hypertree.

Here it is... PPPeople PPPowered version Alpha 0.1 (yes there are bugs and interface issues but it's still lots of fun).

My very first Hyperforest

2010-03-30T08:27:00.000-07:00

So today I decided to look at little deeper at WHAT I want to plot on a Hypertree anyway. It's quite a difficult question. The purpose of all this, ultimately, it to be a browsing experience (not a reporting one) so I don't care if the diagram is an accurate reflection all that matters is that if it is accurate enough to flush out some interesting connections.

The diagram should look plausible but then have oddities in. So.... today I took a simplistic view that went...

What web documents is someone mentioned in
What topics are associated with those documents
What terms are those documents associated with (and which topics)
Who else is on the documents that have those terms

I also threw in some relevancy numbers for good luck ( thanks @TomTague for the tip)

The end results are a quite good...

Gustav Delius - a mathematician who is involved in lots of web projects looks like this...

Trevor Sheldon - the Deputy Vice Chancellor looks like this...

Roger Burrows - Sociologist who might be said to have concerns about technology and privacy (forgive me if I'm wrong Roger)... looks like this...

Sue Hodges - Educator definitely not into social media self-promotion looks like this...

It's looking promising. The pictures look great... now if I can just fathom what they mean I'll be laughing.

Tomorrow I need to research talking to the ZODB and how to make custom object types in Plone so that I can take this data and integrate it into a social networking site we're running.

My Very First Hypertree

2010-03-29T08:36:00.000-07:00

Sometimes, in fact most of the times for me, I find it difficult to work with certain concepts until I can see them. I began today by making the templates that could just show the data I have gathered from Open Calais so that I could get a feel for what's there.

And just now I started adding a little Hypertree magic. Here's some of the data displayed quite simply.

The funny shape makes me realise that I've made a completely arbitrary three-way split of pages, people and terms and that I need to make it build its hierarchy dynamically. Ooh that one hurts.

It looks even stranger when someone happens to have been in lots of pages about meeting minutes. Here's Jo Casey as a Hypertree.

I'm pleased that I'm able to display something. I now need to work on the user interface and add a nice search interface and popup areas for "people information" and maybe to be able to load more data as you click on a node.

I also now need to think about which connections between people are most important and perhaps how to structure the industry terms.

This Week's Plan

2010-03-29T03:32:00.000-07:00

My plans for this sprint are to create a navigable interface for the data Open Calais has returned that focusses on people and connections between people. To begin with this might just be "people pages" or an A to Z page but I also was to quickly start using a few lovely visualisations.

I then want to host this data on a site that people can play with (and into a subversion repository). I will initially use Django simply because I'm very familiar with it but the intention is that this will add data to a Cyn.in (or Plone site) so I have bought a book on Plone development so I can start thinking about saving data to the ZODB...

So that I can start simply and learn how the ZODB works, the plan is to...

Create a crawler for events that saves regular "Events" objects into the ZODB.
Create a new content type for "Places". This may have a lat/long and in the long term should be displayed on a map.... maybe on an iPhone. This is a bit more involved
Create an RSS or RDF grabber Product so that I can integrate people, places, concepts etc with other linkeddata sources... if only to grab an image that represents York the city.

Slight concern: This sort of means that I'll be splitting my direction... one will be looking at visualisation in Django/MySQL the other is looking at data manipulation and integration within Cyn.in (the environment that will ultimately host this data). I'm hoping that once I have got to the end of the visualisation experimentation and the Cyn.in integration work I will be able to pull the two neatly together somehow.

Visualisation Tools

Infovis looks simple enough and perfect for what I need. I particularly like the Hypertree thing (shown below). Here Jeni has taken some government open data about burglary and shown it on a hypertree... brilliant!

The yFiles library has aGraph viewer component but it seems too complete, and it's not as black (which is cool).

Protovis has some very nice display options, I may come back to this later.

Both Raphael and Processing look extremely powerful (go look at the examples) but I will come back to them later once I've done the simple(r) stuff.

One of the lessons learned from the project already is that once you get lots of people working in an open environment that the "Activity Stream" soon resembles a fire-hose of "too much" information. This suggests that the Home Page of any collaborative environment needs to think about displaying aggregate data or visualisations rather than actual data as a way to make the Home Page useful. So it's going to be interesting to try and find novels ways of displaying what for the most part will be conversations.

Sprint 1, it works

2010-03-27T05:14:00.000-07:00

So having crawled the sites, mangled the data... some very simple code...

from djangocalais.models import *

def related_people(name):
   es = Entity.objects.filter(name__icontains=name)[0]
   cds = CalaisDocument.objects.filter(entities=es)
   for cd in cds:
   for e in cd.entities.all():
   if e.type.name == "Person":
   print e.name , ">", cd.__unicode__()

m.related_people("Alastair Fitter")

... produces this....

This is what I like.... Artificial Intelligence for Dummies. I can't wait to slap an interface on this... Now to brush up on some jQuery.

Sprint 1 Done!

2010-03-26T08:53:00.000-07:00

York University's web site has been crawled. The data was then manipulated (or added to) using Open Calais - although I still have to leave that cooking a while longer. And then using the Google Visualisation API I created a tag cloud of what are called the top "social tags" for the semantic information.

The entities returned look really interesting. I need to find out how they relate to each other and how I can find connections between peoples.

The next step is to clean up the code. Make it semi-functional so at very least you can explore the data and then put it into Google Code ... with some instructions! Which will look something like this...

1. Install Django and dependencies

2. crawl (http://www.york.ac.uk)

3. analyze()

4. ./manage.py runserver

What a week...

Python Crawling Tools

2010-03-25T10:47:00.000-07:00

I have been here before, here in this case being in a place where I think, "Surely someone has written a fantastic python crawler that is easy to use and extend that is open source". They may have done, but I can't find it.

My particular problem is this. I want it to be easy to use, something like this would be nice...

from crawler import Crawler

class myCrawer(Crawler):
def handle(self, url, html):
#do_something_here

c = myCrawler(url="http://wherever.com")
c.crawl_type = "nice"
c.run( )

... and also, I'd like it to well designed enough not to run out of memory, well documented, know about the strange unicode formats that live out there on the web, to open too many sockets, to do the right thing when a URL times out... and hey to be quick it probably should be threaded to boot.

After days (and days) of downloading and trying crawlers out there have been some worth noting.

...but I have also tried crawl-e, creepycrawler, curlspider, flyonthewall, fseek, hmcrawler, jazz-crawler, ruya, supercrawler, webchuan, webcrawling, yaspider and more! All of these are rubbish...

The problem is this. It IS possible to write a crawler in 20 or so lines in python that will work... except it won't. It won't handle pages that redirect to themselves, it won't handle links that are ../../ relative, it won't be controllable in any way.

My problem with the two "best contenders" was this. Firstly, HarvestMan, no matter how hard I configure it, always saves the crawled pages to disk. I had this problem in 2006ish, I still have it today. Arnand (lovely guy) the developer has rolled in many of my change requests since then so that the pseudo code above is almost a reality, but only almost. HarvestMan still is a front-runner if just because of the thought that has gone into the configuration options.

Scrapy looked very cool too, but it seems more geared towards getting specific data from known pages, rather than wandering around the web willy nilly. I would have to create my own crawler within Scrapy. I will maybe come back to this but in general, armed with python and a few regular expressions you can't half get a lot done.

So, in my attempt to get "round the loop" once, that is, to a. gather some data from a few sites (namely, the University of York's sites), b. manipulate it in some way (in this case, pump it at Open Calais and see what we get back... poor man's artificial intelligence) and then c.present it ( maybe as a tag cloud, or something more fancy if I have time) I needed a crawler that was very simple to use.

So ...

a. Despite wanting to use an "off the shelf" crawler. I found a crawler that almost worked and hacked it until it worked. It's not threaded, it's not clever but it does the job.

I had to do some hand pruning to look at the mime-types of the pages returned, and remove Betsie pages (of which there were thousands)... but I will try and roll that back into the crawler.

b. I found a Django application called django_calais, which after adding the last line to my Page model, like this...

class Page(models.Model):

url = models.URLField(unique=True, null=False, blank=False, db_index=True)

title = models.CharField(max_length=300, null=True, blank=True)

crawl_date = models.DateTimeField(default=datetime.now)

html = models.TextField()

type = models.CharField(max_length=100, null=True, blank=True)

size = models.IntegerField(null=True, blank=True)

calais_content_fields = [('title', 'text/txt'), ('url', 'text/html'), ('html', 'text/html')]

... I could then run....

def analyze():

pages = Page.objects.filter(type="text/html")

for page in pages:

print page.__unicode__(), page.url

try:

CalaisDocument.objects.analyze(page, fields= [('title', 'text/txt'), ('url', 'text/html'), ('html', 'text/html'),])

except Exception, err:

print err

.... and have my Calais application be populated with People, and Organisations and Companies and Facilities etc. all of which are related back to my Page model.

I haven't got to presentation stage yet, the Calais analyzer is still running, BUT after get quite anxious about spending too long looking for an adequate crawler, the semantic bit has already proven itself. So maybe tomorrow I will be able to present some data...

And then I'd better check it in to a repository or something.

Review of Python Crawling Tools

2010-03-22T05:53:00.000-07:00

http://www.ohloh.net/p/WenChuan

Web crawlers are funny things. I have had a little experience of working with web crawlers in the past with a mixed set of results. Many were too difficult to configure or simply not robust enough. At this stage of the project, I would ideally like a crawler that is easy to adapt and yet trust-worthy enough to be able to set it running and leave it.

Imagining that I don't want to crawl very large portions of the internet, instead looking just to work with small corners I will be happy with something simple. Worth trialling are alternatives to "doing it yourself", such a Yahoo Pipes and the 80legs.com crawling service as well as Yahoo Boss search engine builder. Whenever possible I will consider using these tools, simply because of the time needed to crawl for data.

My preferred language is python but I wonder if there is a more web-centric language to think about? I vaguely remember Rebol as having URLs as base types. Maybe Rebol is worth exploring.

I started by looking at a list of python crawlers at Ohloh http://www.ohloh.net/p?sort=users&q=python+crawler and from these tested these....

Testing Notes...

Ruya (Not bad, has before_crawl and after_crawl functions that you overshadow)

Mechanize (Kind of browser simulation - including form filling)

import re
from mechanize import Browser

br = Browser()
br.open("http://www.example.com/")
# follow second link with element text matching regular expression
response1 = br.follow_link(text_regex=r"cheese\s*shop", nr=1)
assert br.viewing_html()
print br.title()
print response1.geturl()

Crawl-e ( Not bad )
The CRAWL-E developers are very familiar with how TCP and HTTP works and using that knowledge have written a web crawler intended to maximize TCP throughput. This benefit is realized when crawling web servers that utilize persistent HTTP connections as numerous requests will be made over a single TCP connection thus increasing the throughput.

#Squzer - missing in action

SuperCrawler - not bad but the code looks a bit terse.

Fseek
Fseek is a python-based web crawler. The user-interface is implemented using Django, the back-end uses pyCurl to fetch pages, and Pyro is used for IPC.
Says it's Django but it's not. I liked the idea of it being "presentation ready"... could be handy.
   cannot create writable FSEEK_DATA_DIR /var/fseek.
   ImportError: No module named Pyro.naming
   failed to load external entity "/var/fseek/solr/etc/jetty.xml"
   Needed Pyro

Webchuan ( XPathy)

WebChuan is a set of open source libraries and tools for getting and parsing web pages of website. It is written in Python, based on Twisted and lxml.

It is inspired by GStreamer. WebChuan is designed to be back-end of web-bot, it is easy to use, powerful, flexible, reusable and efficient.

Error in the setup.py script removed the description and it worked.

Jazz crawler
   Broke on unicode data (added any2ascii wrappers and it solved it -- hack)
   Then borked on a draw_graph (probably too big)
    Interesting because it builds a graph (and calculates PageRank) of the data gathered enabling an out-of-the-box simple visualisation...

Harvestman
I have used this before and found did "too powerful", I never did work out how to get it not to save crawled files to disk. I was unhappy with the config.xml approach to running the crawler too. If Scrapy proves unsuccessful I will return to this because it's a great product.

ExternalSiteCatalog - an integration of Harvestman with Plone - may be useful later.

AWCrawler - a python spider that saves data to Amazon S3 etc

Scrapy (XPath and pipelines)

Scrapy is new (to me) and looks easy to modify. I will experiment with this, though will need to learn some XPath to begin with.

The Sprint Cycle

2010-03-22T05:09:00.000-07:00

So that I don't "bite off more than I can chew" -- my aims are to make many iterations of the Sprint Cycle (at a guess, four-ish). The dotted areas on the diagram above show the gaping holes in my knowledge. The first iteration will be about getting a better understanding of each of the areas and finding the simplest tools for working with them.

Sprint 1: The aim to to produce the simplest thing possible that is remotely useful. I will attempt (this week) to make a full traversal of the cycle. At this point I don't have the details of people's profile on social media sites (the survey hasn't been published yet), I don't have any real knowledge about worth with repositories, nor of working with LinkedData. I will need to work with what I have, namely the data that is already "out there"... web pages and links.

Whilst the aim may be to make something more like the Research Portal (shown below), this week is going to be all about doing something MUCH SIMPLER.

Aims

Review crawling components (and mine York's existing data - wherever it is)
Display content in a TagCloud
Attempt to integrate with some LinkedData
Attempt to present and data or relationships in a novel way
Publish the Social Media Survey to hopefully find out more about what is being used at York.

The first iteration of the cycle looks like this. And nothing like thrashing around in frogspawn.

Milestone 1 Report

2010-03-22T04:49:00.000-07:00

I have trialled a number of tools (SocialText, Jive, Elgg, Confluence, LifeRay Social Office and Cyn.in) with a number of teams which include...

Social Policy Research Unit
Research Computing Opportunities
Sustainability Forum
Science And Technologies Studies Unit
eLearning Development
Computing Service
Collaborative Tools Project
Marketing Interest Group
Humanities Research
Liaison Librarians
Stockholm Environment Institute
Library
Planning
PURE Project
Complex Systems
Philosophy
Employability Co-ordination Group
18th Century Studies
Business Analytics
... and although we now have 117 registered members, many of those are not particularly active... BUT many more are contributing to the discussion about the needs and directions of the project from a mixture of background, research, development, management, support and financial. The important thing is that the debate surrounding the project is happening in the project tool itself, and not hidden on email lists or password protected wikis.

The Social Media Survey is complete and awaiting publication. This will hopefully find sites that we don't know exist and the extent of the use of different tools at the university.

Milestones

2010-03-18T04:54:00.000-07:00

Milestone 1: People and Site

a. People. The first milestone is about assembling a team of people engaged with the as yet, non-existent PPPeople PPPowered technology. This team will provide the data, particularly their social media usage and profiles etc with which the initial data-mining can be built upon.

Goals: To have at least 20 people in a number of teams of people willing to work with me on the JISC project, providing hard data, usage and feedback.

b. Presentation. The data that is ultimately mined needs to be "hosted" in a software tool that provides a level of service that is valuable enough to be used frequently. This will ensure that meaningful relationships between people are presented and then subsequently pruned by participant. The choice for which tool we use at this point was initially between Wordpress, Drupal, Elgg and LifeRay.

The conceptual framing of what this tool should be, a people directory, a people discovery tool or a profile repository or a personal new aggregator is quite important with regards to setting expectations (for re-visiting the site)

Outcomes:

To have a site up and running, available to staff at University of York that at least displays their profile in some way.
A blog post reporting on the project so far, the plan, the tool chosen (with reasons) and invitations to help with the plan (which semantic repositorities to us (and how), which crawling tools to use
Review the approaches take with other social networking sites to gather any useful approaches. For example, are their benefits to be gained by leveraging an existing social network and evangelising the use of a certain social network in order to ease data gathering.

This phase is very dependent on peoples' time availability and willingness to engage and also on the richness of the data returned from the survey. If nobody at all uses social media or completes the survey we won't have a lot to begin with.

Milestone 2: Just Data and Display

The first stages of displaying mined data with avoid the complexities of semantic reasoning about the data, about using unusual sources of data and begin by simply asking people to complete a survey. This will hopefully result in a spreadsheet of sites, blogs, social media memberships (twitter, linkedin, CiteULike etc) and will form the basis for exploring slightly less explicit data (connections in Linked in, followers on Twitter, mentions in Twitter or on other people's blogs.

To display this data, initially we will need to adapt the profile in the presentation tool, perhaps including an RSS aggregation.

Outcomes:

A survey asking people to identify their social media accounts, interests in the form of keyword, URLs etc
A site with at least 20 members showing their "simple" social media membership and some of the data (recent tweets, friends etc).
The initial adaptation code (module/plugin) uploaded to the SVN site.
A blog post showing developments and with comments from members.

This will require a good understanding of the plug-in architecture of the presentation tool. The whole point of this project is to be integrated into a tool that is usable and used.

Milestone 3: More Deeply Mined Data

This stage looks to find information beyond that that is given, perhaps crawling Google searches, finding more distant links between resources. Ideally any crawling or querying tools should be well integrated into the presentation tool OR componentized and talk to the presentation tool via XMLRPC or similar.

We will experiment with (probably python-based) web-crawlers such as Domo, Harvestman, Mechanize and Scrapy. In addition we will look at free or low lost tools and services available. These may include...

Yahoo Pipes - online data manipulation and routing
80 legs - a crawling service
Maltego - a desktop open source forensics application
Picalo - desktop forensics application (or any other tool)

Outcomes:

More interesting data, richer connections
A generic crawling methodology

Whilst I am more than familiar with creating simple crawlers this will need a more standalone, robust, better architected approach.

Milestone 4: Semantified Data

This stage will look at how to apply semantic tools, or a more reasoned understanding of the data gathered. We will be looking at ...

Yahoo Boss for general searching.
OpenCalais, to attempt to discover entities in unstructured data
DBPedia, Edina, Freebase for understanding entities like Towns, Universities, Concepts etc
ePrints research repository to connect people via research outputs

SPARQL is very new to me. I need more understanding of RDF, LinkedData etc. Although the JISC Dev8D conference gave me new insights into the possibilities presented by LinkedData and open data I still feel I have a way to go to fully understand this area.

Milestone 5: Slightly More Sophisticated Presentation

Depending on the data gathered, there will be opportunities to present the data in more interesting and usable ways. Initially we will attempt to use simple visualisations such as timelines, tree maps, word clouds etc. which can be easily integrated in the presentation tool.

More ambitious visualisations such as Mention Map example will be explored if appropriate.

Need to understand more about the maths behind networks and visualisation. Luckily Gustav Delius is around to advise.

Basic Plan

2010-03-16T10:10:00.000-07:00

This shows the basic plan for the project. The hope is that the first run of this process should be complete as soon as is possible. This means that the REASONING and VISUALISATION stage may be omitted, where the the data can be presented as a simple list.

This makes the choice for the DATA-GATHERING and PRESENTATION tools very important. We want them to be very well integrated. Ironically, given the importance of the PRESENTATION stage, namely what user profile data it stores (social media usernames etc), what features it supports (RSS aggregation, Twitter presentation, Plugin development opportunities etc).

Reflections on Tyre-Kicking

2010-03-06T07:30:00.000-08:00

Creating Teams of "Tyre-kickers"

2010-02-16T07:30:00.000-08:00

Collaboration is impossible to fake

One of the problems with evaluating collaboration tools is that they can't be compared on features. One tool's blog can be very different from another tools blog. A small missing feature from a wiki can make it less easy to use.

The only real way to test which collaboration tool will best suit the PRESENTATION layer, the part where any mined data is displayed and used is to use the tool with a number of groups of real, live, collaborating people. With this in mind I have tried to engage a number of teams willing to "throw their all into testing an environment"... their nickname is "Tyre Kickers".

A large component of this projects success will not depend on whether or not the technology works but whether or not people use it.

One of the problems with this Tyre-Kicking approach is that once people have invested time and effort into working with, and more importantly around a tool, they may have grown to like the tool despite its warts and be loathe to move on to another tool because all their content exists somewhere else.

Aware that the different teams will have differing uptakes in terms of engagement, some will want and need to use the tools whereas others will dabble. Some groups will work in the same office whilst others will be dispersed, across and beyond the university.

The Tyre-Kickers I have held meetings with and introduced to up to three tools currently are....

Applications Deployment - A Computing Service team
Biology - Development Team
Collaborative Tools Project
Communications - Small departmental team
Computing Service - Large departmental team
Court, Country, City. British Art 1660 - 1730 - Humanities project team
Digital Library - Development team
Finance at University of York - Small department
Humanities Research Centre - Large cross disciplinary departments
IT Support Office - Small team
Liaison Librarians - Large dispersed team
Marketing Interest Group - Dispersed team
Mathematics - Small team
Planning at University of York - Small team
Research Themes - Small project team looking at strategy
Science and Technology Studies Unit
Sociology - Small team
Stockholm Environment Institute - Large collaboration team
Sustainability at University of York - Small evangelisation team
Tourism and Topography in Britain - Humanities project team
Web Office - Small departmental team
Wireless Strategy - Small computing service team
York Centre for Complex Systems Analysis - Large cross departmental team
YorkShare HQ Revamp - Small development team

In some of the cases above, some have barely used the tools, some already have some systems in place whilst others have if anything being using the tools too much, investing lots of time and effort in adding and linking content.

The Tools

2010-02-02T07:31:00.000-08:00

List of tools to be used

The Collaborative Tools Project

2010-01-16T10:30:00.000-08:00

The Collaborative Tools Project aims to improve collaboration at the University of York. Currently York has no central intranet system and no real support for academic blogging. A dozen or so MediaWiki wikis exist for specific technical teams (those who knew who to ask) but this was rightly seen as not an option going forward in terms of scalability.

As a result, understandably, people have gone off-piste as it were and made use of Wordpress, Blogger, Google Docs and the like, or are running their own servers with tools like Wordpress, Moodle and Drupal.

The Collaborative Tools Project looks to offer blogging, instant messaging, wikis, rich media hosting, discussion tools for all staff at the university. It is not looking to work with students because the requirements for student-facing systems are very different.

Work began a few years ago gathering Use Cases to discover what both academic and other staff at York needed. These can be distilled down to...

Collaborative Document Editing - working on documentation, bid documents, research papers ect
Project Workgroup - private wikis, blogs, file-sharing etc
Public Presentation - public-facing blogs, media rich, communities of practice, aesthetically pleasing
Other Stuff - meeting support, sharing minutes, gathering consensus, broadcasting

How The PPPeople PPowered project fits with the Collaborative Tools Project

Interestingly or perhaps predictably, these Use Cases threw up what people wanted and needed but little in the way of what might improve the information eco-system at the university. Almost nobody came to the Collaborative Tools Project with needs that were community focused.

This JISC project will create a people-oriented, browsable area of discoverability where people will simply be able to find out about each other, their work and interests. Whatever tools (blogging, wikis etc) we provide for the staff will integrate with the PPPeople project acting as a semi intelligent hub.

Because the PPPeople project is embedded in a larger project, it guarantees plenty of real users providing feedback and use cases during its development.

Brief Project Description

2010-01-09T07:29:00.000-08:00

The aims of this project are to help enable collaboration at the University of York by making people and their work more easily discoverable and more richly linked. Currently at York the people directory is very basic (shown below) revealing nothing of a person's work, interests, colleagues or personality.

This project looks to create richer connection between people using...

mined data - such as web pages containing their email address or blog posts or Google queries
known data - such as research repositories, such as CiteSeer, Mendeley, ePrints at White Rose Consortium or CiteULike
known organisational data - such as what department somebody belongs to
social media usage - such as Twitter followers, LinkedIn Connections and Delicious tags

Above is a wireframe showing a simplification of the sorts of elements we expect to be able to present, including tags, people, links and related articles.

The Caveats

Of course, it would be easy to believe that creating an Amazon-esque recommendation list (below) is easy. Creating these sorts of tools can be deceptively difficult with the development always on the brink of a breakthrough and never quite able to fully deliver.

Having experience of these pitfalls this project will initially aim to implement the simplest technological solution available. We will begin by looking at the known, later extending towards the reasoned and guessed connections. We will begin with explicit connections and later explore semantic relationships.

Rather than beginning with the technology we will simply ask people which social media tools they use with which to pre-populate our knowledge of people. This will take the form of survey. In addition to asking people for their social media usage the tool should allow people to explicitly make connections.

So that this tool doesn't become a technological folly, presenting beautifully visualised connections between people but never actually being used, the presentation of this data and these connections needs to be embedded in an everyday online environment. By that, I mean that this tool should not be a destination in its own right but should be stumbled across naturally whilst looking someone up or engaging in discussion.