Tuesday, October 22, 2013

Recap: GraphConnect SF 2013

The Driver Writers
Our fearless leader Emil Eifrem presents the new Neo4j web browser
Sessions packed with attendees
GraphClinicians assist attendees with their graphDB problems
GraphClinician Amanda Laucher helps with a proof of concept
BBQ lunch while enjoying SF's Indian summer
(node_3) < - [:CONNECT] - (node_2) < - [:CONNECT] - (node_1)
Wow, GraphConnect SF 2013 was awesome! We had almost 400 people attending the conference and we would like to say a big thank you to all of the attendees, over 23 speakers, staff and other participants who made it such a fantastic success.

Before the conference, we brought in the authors of Neo4j language drivers from around the world for two days of intense development work at the inaugural Neo4j Driver Authors Hackathon. Many met for the first time and were able to dedicate time to bring the Neo4j drivers up to speed with the Neo4j 2.0 release and the developments in the data model (Labels, Indexes), Cypher HTTP endpoints, Neo4j-Browser and other interesting things.

During the day on Thursday, we held a full day of trainings with four completely sold out courses: Introduction to Neo4j, Neo4j in Production, Data Modeling and Import and Advanced Cypher Queries.  It was great meeting newbies to graph databases as well as enthusiasts taking more advanced Neo4j courses.

The conference sessions began on Friday with a keynote by Emil Eifrem titled "New Opportunities for Connected Data" (click here to watch the recorded video). The room was packed as Emil spoke about our new Neo4j browser interface, available in Neo4j Milestone 2.0.0-M06, and an upcoming online training.

The morning continued with presentations in three different tracks with talks on topics like Neo4j internals, graph visual analysis, graph databases in gaming and using graphs to manage and mitigate risk.

We also hosted our GraphClinic, where attendees could bring their graph questions and issues and get help from a Neo4j senior consultant. Our GraphClinicians Max, David and Amanda had a fantastic time providing diagnoses and prescriptions.

Next up was our BBQ lunch. Attendees received blankets at registration, so they could enjoy the picnic on the grass in the glorious sunshine!

Fred Trotter presented a keynote titled "Hacking Healthcare with Neo4j" which covers his work with DocGraph, a graph of how healthcare providers collaborate in the U.S. and probably the largest named graph publicly available. His talk (available here) demonstrates the power of graph databases to handle such complex and connected data.

The day continued with more talks on a variety of graph database topics like modeling graphs for healthcare, telecommunications, career paths and beer. Our Developer Evangelist team hosted an unconference that featured lightning talks from community members. During the afternoon, we also organized a GraphPub so that GraphConnect attendees could mingle with Neo4j developers over games and beer like in this picture.

The conference closed with Jim Webber delivering a final keynote titled "A Little Graph Theory for the Busy Developer" (available here). We moved upstairs for a cocktail reception and then right back downstairs for our 2nd Annual Graphie Awards Dinner. The Graphies recognize and celebrate individuals and teams that are developing innovative and impactful graph database applications. Click here for a list of this year's winners.

To watch recorded videos from GraphConnect SF 2013, check out graphconnect.com/video

Wanna see pictures from the conference? Head over to our Flickr.

Be sure to join us at our upcoming conferences:

November 5-6, 2013
November 18-19, 2013

Below are some of our favorite tweets from GraphConnect SF, but check the @GraphConnect and #GraphConnect pages to see what other people thought of the conference.

Tuesday, October 15, 2013

Neo4j 2.0.0-M06 - Introducing Neo4j's Browser

Type in a Cypher query, hit <enter>, then watch a graph visualization unfold. Want some data? Switch to the table view and download as CSV. Neo4j's new Browser interface is a fluid developer experience, with iterative query authoring and graph visualization.

Available today in Neo4j 2.0.0 Milestone 6, download now to try out this shiny new user interface.

Cypher Authoring

Neo4j Browser is like a mashup of a REPL and a lightweight IDE, sprinkled with pretty graphs (ooh, ahh). Writing and running Cypher queries is the center of all the action, which happens in the editor along the top. Just type in a single-line query and hit <enter>. You'll get a result frame in the stream, showing either a table of property data or a graph visualization of nodes and relationships.

Type another query, get another frame. Tap the up arrow to retrieve a previous entry, edit, then run it again.

For larger queries, hit <shift-enter> to switch into multi-line editor mode. Now you'll need to use <ctrl-enter> to run, and <ctrl> up or down arrow to navigate history (the modifier also works in single-line mode).

Finally, and thankfully, you can save scripts. Hit the star button to save the current editor content, which will be available in the sidebar. By convention, the first line can be a comment which will be used as the name of the query. You can even drag-and-drop in scripts, for sharing queries or small-scale data import.

Query result visualization

To show your proud parents (or maybe boss) the work you've been doing, nothing beats graph visualization to explain what's going on. All exploration starts with a Cypher query that returns nodes or relationships.

Click on any node or relationship in the visualization to bring up an inspector which will reveal its properties. Switch to the inspector's styling tab to tweak the colors and content, based on node label or relationship type.

Developer's Friend

A primary design goal of Neo4j Browser is that every interaction is a demonstration. It should be capable enough to get things done, while recognizing that your main interest is developing an application. Any "buttons" which do anything work by running queries or commands which you can learn from.

All the actions in the sidebar works like that, using client-side commands to run queries, interact with the REST api, or load content like help pages and mini guides. The interface should do things for you, but also show you how things are done so you can smoothly transition into application development.

Result frames adopt that perspective by being both meaningful and useful: examine the result data, then click a button to download raw JSON results or a CSV table.

We've got plans for extending this approach to make this a tool which integrates with your everyday workflows, whether debugging, prototyping, or exploring. Got some ideas of your own? Let us know how you use Neo4j on a daily basis.

Your turn

Neo4j Browser debuted at the recent Graph Connect in San Francisco, with a live preview during the keynote.  You can get it today.

Thanks to everyone who helped beta-test the new interface.


Monday, October 7, 2013

The Neo4j Driver Hackathon at GraphConnect San Francisco

Hi all,

we are all still totally stoked after 2 days of intense development work at the inaugural Neo4j driver authors hackathon. Thank you all for joining us in San Francisco and the San Mateo Neo Technology office and sorry Michael Klishin that it didn't work for you to come! 

Many of us saw each other for the first time, and it was a perfect kick-off to GraphConnect San Francisco on Friday.

With the invitation we wanted to extend a small thanks to some of the most prolific members of the Neo4j community. 

Being together, we hoped to dedicate some time to bring the Neo4j drivers up to speed with the Neo4j 2.0 release and the developments in the data model (Labels, Indexes), Cypher HTTP endpoints, Neo4j-Browser and other interesting things.

We also had some great discussions on the Neo4j roadmap with Philip providing feedback from some of the most insightful Neo4j users worldwide.

Aseem, we're sad you got ill, we hope you get better soon, impressive progress anyway!

The Hackathon attendees:
Back row: Nigel Small, Tobias Lindaaker, Max DeMarzi, Jason McAllen, Stefan Armbruster, Michael Hunger, Tatham Oddie, Andreas Ronge, Elad Olasson, Philip Rathle, Javier de la Rosa
Front row: Josh Adell, Aseem Kishore, Peter Neubauer, Matt Luongo, Wes Freeman

The driver progress summary

It’s been a productive 2 days, let’s sum up the progress we made:

Michael Klishin

Nigel Small

Andreas Ronge
  • The neo4j-core Ruby gem (used by neo4j.rb gem):  a simple unified api now works in both JRuby via the embedded Neo4j API and via the transactional Cypher server endpoint
  • Improved transaction support using the Neo4j server transaction endpoint.

Stefan Armbruster

  • started work on a new, Neo4j 2.0 compatible version of the Neo4j Grails plugin
  • new plugin version will use Cypher only and therefore performance is exptected to be much better when used remotely
  • so far, 8 out of ~35 test classes of the GORM TCK are passing

Javier de la Rosa (neo4jrestclient, Python)

  • updated tests to make them work with the last 2.0.0.M06
  • removed the old syntax for creating relationships based on neo4j.py
  • dropped support for Python 2.6 and prepared the path for Python 3.3
  • implemented full support for labels (but not still for batch operations)
  • started the work for Cypher transactions
  • started the work for .filter on labels that run Cypher queries underneath

Aseem Kishore (node-neo4j)

  • caught up on outstanding bugs and feature requests, particularly around indexing
  • started work on Neo4j 2.0 support (transactional Cypher endpoint, and labels)
  • begin planning node-neo4j 2.0 (rethinking of API, esp. around Cypher-only)

Wes Freeman (AnormCypher)

  • released 0.4.3 (supports 2.0 minus transactional), integration tests against M05

Peter Neubauer

  • updated the Neo4j Geoff plugin to Neo4j Server 2.0 and fixed the tests

Michael Hunger

  • updated the Neo4j JDBC driver for Neo4 2.0

Tatham Oddie (the token Windows guy):

  • Neo4jClient for .NET
    • Added support for newer Cypher clauses that have shown up in Neo4j 2.0
    • Made good progress on supporting transactional Cypher within the standard .NET convention of using (new TransactionScope) {}; some final scenarios to support before I land the branch
  • Prototyped Cypher integration in Excel: https://twitter.com/tathamoddie/status/385833088091041792
  • Worked on improving Neo4j on Windows, with the aim of replacing the ageing .bat files with .ps1 equivalents. The first step was working out how to resolve Java on Windows with any degree of reliability: https://gist.github.com/tathamoddie/6817880

We really hope that we can do this every year from now on - join the team and start hacking!


Wednesday, October 2, 2013

The first GraphGist Challenge completed

We're happy to announce the results of the first GraphGist challenge.
Anders Nawroth
First of all, we want to thank all participants for their great contributions. We were blown away by the high quality of the contributions. Everyone has put in a lot of time and effort, providing thoughtful, interesting and well explained data models and Cypher queries. There was also great use of graphics, including use of the Arrows tool.
We thought we had high expectations, but the contributions still exceeded them by far.
In this sense, everyone is a winner, and we look forward to sending out a cool Neo4j t-shirt and Graph Connect ticket or a copy of the Graph Databases book to all participants. And for the same reason, we strongly advice you to go have a look at all submissions.

Here are all the contributions:
As you can imagine, we had a hard time deciding which contributions should get the first, second and third price. Anyhow, here's the result, in reverse order:

Third Prize

At third place, we find Chess Games and Positions by Wes Freeman. He makes it all sound very simple:
The goal is to load a bunch of chess games into Neo4j for further analysis. Scores listed are Stockfish’s take on a position after a 25 move horizon (but this number can be deepened as the graph is filled out or as more processing is done). Positions can also be loaded as alternative moves (not connected to a game) based on suggestions from Stockfish. The positions are recorded as FEN, a human-readable/compressed chess board state notation.
And the data model is not overly complex at all, here's a bit of example data:

We thought GraphGists have quite much interactivity, but Wes shows how to get even more interactivity into a GraphGist. After simply listing the moves of a game, he goes on to show off some cool statistics, which reveals the blunders in a game and even suggests better moves.

Second Prize

Learning Graph by Johannes Mockenhaupt comes in at second place. Here's his own introduction to it:
This graph is used to visualize the knowledge a person has in a certain area. … The purpose is to document acquired knowledge and to help to further educate oneself in a structured way. This is accomplished by graphing dependencies between technologies as well as resources that can be used to learn a technology and to determine possible learning paths through the graph, which show a way to learn a specific technology, by first learning the technologies, in order, which are prerequisites for the technology to be learned. The graph is meant not to be static, but updated as new connections between technologies are discovered and new knowledge is acquired.
This is how the data model plays out with a tiny set of data:

The data model is easy to grasp, and at the same time, it shows the power of graphs in a prominent way. The queries are surprisingly simple — if you ever tried to do something similar using an RDBMS, you'll appreciate the straightforwardness and elegance of the queries presented! It's also nice to see how the data gets updated along the way. Finally, the explanations of the queries and their results binds everything together to form a pleasant read.

First Prize

The US Flights & Airports contribution from Nicole White finished first in this challenge. Congrats Nicole!
Here's the background:
For any airline carrier, efficiency is key: delayed or cancelled flights and long taxi times often lead to unhappy customers. Flight planning is one the most complex optimization and scheduling problems out there, requiring a deep analysis of flight and airport data.
A simple proposed data model which allows complex questions to be answered. One of the strengths of a graph database. The interesting details were not in just modeling the flights but also the cancellations and delays.

Nicole stated interesting questions on top of the data model and dataset which she was going to answer using Cypher queries:

  • What is the average taxi time at each airport for both departures and arrivals?
  • What is the leading cause of departure delays at each airport?
  • How many outbound flights were cancelled at each airport?
Or more specific questions such as:
  • Which flights from Los Angeles (LAX) to Chicago (ORD) were delayed for more than 10 minutes due to late arrivals?
  • How does seasonality affect departure taxi times at Chicago’s O’Hare International Airport (ORD)?
  • What is the standard deviation of arrival taxi times at Dallas/Fort Worth (DFW)?
To show just one example:

Which flights from Los Angeles (LAX) to Chicago (ORD) were delayed for more than 10 minutes due to late arrivals?

MATCH (a)<- data-blogger-escaped-f="">(b), (f)-[r:DELAYED_BY]->(d) WHERE a.name="Los Angeles International Airport" AND b.name="O'Hare International Airport"       AND r.time > 10 AND d.name="Late Aircraft" WITH f, r.time AS latedelay RETURN f.flight_number AS Flight, latedelay AS `Delay Time Due to Late Arrival`

This query results in:
FlightDelay Time Due to Late Arrival

With her scientific approach, listing included variables and using MathJax to render the used mathematical formulas, this submission is really impressive and a worthy winner.

Our congratulations go to every participant and the winners. We are really thrilled about the results of this competition.

GraphGists evolving & The next GraphGist Challenge

During the challenge we improved the code behind GraphGists:
  • We added support for Math formulas.
  • We added Disqus integration, so there are now comments connected to each GraphGist. Please add your comments to the challenge contributions, the authors will be happy for feedback and suggestions.
  • We removed the annoying headings above result tables and graphs.
  • We fixed some issues and added a workaround so Chrome under Windows doesn't crash.
  • We improved the styling a bit. (It's still very primitive though.)
Thanks for everyones feedback: it helped us iron out some of the shortcomings.
If you want to have a look at the GraphGist project, it's located here: https://github.com/neo4j-contrib/graphgist. It's a client-side only browser-based application. Meaning, it's basically a bunch of Javascript files. We'd be happy to see Pull Requests for the project. Please note that you can contribute styling or documentation (as a GraphGist), not only Javascript code!

We already got questions about the next GraphGist challenge. Our plan is to run the next challenge around the time Neo4j 2.0 gets released. Currently we think that will mean a closing date before Christmas. We'll keep you posted when we know more.

Greetings from the Neo4j GraphGist Challenge gang!
Anders Nawroth, Peter Neubauer, Michael Hunger, Pernilla Lindh, Mark Needham, Kenny Bastani