Thursday, July 18, 2013

Neo4j 1.9.2 now available!

Hello fellow graphistas,

Today we're releasing the latest version of the 1.9 series of Neo4j, version 1.9.2. While there aren't any new API-level features in this release, there are plenty of goodies under the covers.

Firstly we've been working with the lovely folks at ThingWorx to optimise the way Neo4j performs IO when deployed on Windows. The 1.9.2 release can substantially reduce the amount of IO that Neo4j performs in some cases when running on Windows.

Then there's a handful of tweaks and bug fixes that improve the overall experience of running and querying Neo4j, including useful improvements to the way we set up networking for HA clusters and the documentation for the same.

Finally there are some fixes to the REST API to ensure we're using the Content-Encoding header properly (thanks to Nigel Small) and to fix a potential problem whereby JSON can become mangled coming out of the REST API (thanks to Wes Freeman).

Neo4j 1.9.2 is available immediately and is an easy upgrade from any other 1.9.x versions as there are no store upgrades required and so everyone on Neo4j 1.9 and Neo4j 1.9.1 is strongly encouraged to upgrade to 1.9.2.

You can download from the neo4j.org web site.

Until next time, stay connected.

Jim Webber
Chief Scientist @ Neo Technology

Saturday, July 13, 2013

Fun with Music, Neo4j and Talend

Many of you know that I am a big fan of Belgian beers. But of course I have a number of other hobbies and passions. One of those being: Music. I have played music, created music (although that seems like a very long time ago) and still listen to new music almost every single day. So when sometime in 2006 I heard about this really cool music site called Last.fm, I was one of the early adopters to try use it. So: a good 7 years later and 50k+ scrobbles later, I have quite a bit of data about my musical habits.

On top of that, I have a couple of friends that have been using Last.fm as well. So this got me thinking. What if I was somehow able to get that last.fm data into neo4j, and start "walking the graph"? I am sure that must give me some interesting new musical insights... It almost feels like a "recommendation graph for music" ... Let's see where this brings us.

Basically, the approach I took had three simple high-level steps

  1. get the data from last.fm
  2. model that data into a neo4j graph
  3. pump the data into the neo4j database using an import tool
  4. query the data for hours on end ;)
So let's get right to it.


Step 1: exporting the data from last.fm.

Turns out there are some cool tools out there to get the scrobble data out of last.fm. I used the LastToLibre "lastexport.py" script, which is very easy and simple: run it, give it a user name, and the public scrobbles will be available shortly after in a text file with the date, trackname, artistname, albumname and then the "MusicBrainz" identifiers for the track (trackmbid), artist (artistmbid) and album (albummbid). I did this for myself and two friends, and got a sizeable dataset.

Step 2: create a model out of this.

From this dataset, I then had to create a neo4j graph model. As you know, there are multiple ways you could model the data - it really depends on the query patterns - but after some consideration I went with this model below:


Ok, so I got the data, and got the model - how do I now get it into neo4j?

Step 3: import the data

This is where it got interesting. The spreadsheet import mechanism worked ok - but it really wasn't great. It took more than an hour to get the dataset to load - so I had to look for alternatives. Thanks to my French friend and colleague Cédric, I bumped into the Talend ETL (Extract - Transform - Load) tools. I found out that there was a proper neo4j connector that was developed by Zenika, a French integrator that really seems to know their stuff.

So on I went. Installed the Talend Open Studio for Big Data, which is Open Source Software, just like Neo4j, and started playing around. I found the tools quite intuitive - although I must admit that Cédric was a great help to show me around. All I had to do was to created a Talend job, which consisted of a couple of steps:
  • Import the nodes: 2 subjobs, one for nodes with name and type, one other for the nodes that have name, type and a "Musicbrainz Identifier" (artists, tracks, albums),
  • Import the relationships: 7 subjobs, one for every relationship type (see model). Important there is that there is an additional step here, which is to make the relationships "unique", and avoid that some relationships would be created twice.

It was very interesting to see that the import process only took about a minute with Talend, versus more than an hour with my Excel/neo4j-shell method. And if I would use the "batch import" mechanism (which does not commit transactions) it would probably be even faster. Here's a little overview video that shows you how to do it step by step - it literally only takes 10 minutes to do.



Step 4: do some neo4j Cypher queries

After the talend job was done, I started to experiment with some Cypher queries: figure out which artists my friends had been listening to on the same day: 

START
    sta=node:node_auto_index(name="STA"), 
    sno=node:node_auto_index(name="SNO")
MATCH 
    sta-[:LOGS]->scrobble-[:ON_DATE]->date, 
    scrobble-[:FEATURES]->track<-[:PERFORMS]-artist, 
    sno-[:LOGS]->scrobble2-[:ON_DATE]->date, 
    scrobble2-[:FEATURES]->track2<-[:PERFORMS]-artist
RETURN 
    distinct date.name, artist.title;

Seems like these queries are not that trivial, and there probably still is quite a bit of optimisation to be done - but that's way above my capabilities. And obviously there are many more ideas for interesting queries - the music domain is very graphy in nature, and allows for more hours of graph fun. But that will be for a later time.

Hope this is useful! Enjoy the summer!

PS: