Friday, November 25, 2011

Neo4j 1.6.M01 “Jörn Kniv”

Hi there,

We have a new milestone for y’all. Previously we’ve had “Boden Bord” and “Abisko Lampa”, both going southwards through Sweden from the very north. And now we’ve released the first milestone of the up and coming 1.6, named after the small city of Jörn.
Before writing about it, I’d like to get a quick chance to introduce myself (this feels like stealing the stage...). My name is Björn Granvik and I’m based on a true story. A developer at heart and soul and in charge of engineering here at Neo Technology. I sometimes end up thinking out loud here: http://bjorngranvik.wordpress.com/ Nuff on that.

The theme of 1.6 is mainly about improving infrastructure and QA. These improvements include faster builds, moving from TC to Jenkins, and extending our tests to cover more client platforms, both browser and operating system wise. The reason for these changes is that, while we've delivered many great features very rapidly over the last few months, we’re always looking to do better. Improving our internal build infrastructure helps us deliver quality features faster, and helps us better turn around responses to the community's requests for features.

Infrastructure isn’t our only focus for 1.6, however. We are also working on Neo4j so that we can store graph metadata, e.g. configuration settings. This will help us to better evolve the internal infrastructure.
As always, there are a number of bugs that have been fixed, both internally and for the community issues. See: https://github.com/neo4j/community/issues?sort=created&direction=desc&state=closed&page=1

Pablo Pareja Tobes, maker of http://www.bio4j.com/ started a community poll http://www.doodle.com/wg8k77vwq6b654bv which we are taking into consideration when moving forward with the next milestones. Thanks for the feedback!

Keeping up with the features (and bugs) in Firefox This milestone does not support the new Firefox8 and Firefox9 releases, since we need to work around some issues with webworker support being removed - stay tuned for that.

/Björn

PS: “Jörn” is a place in north of Sweden. “Kniv” is knife in English. Sort of makes me wonder from where the English got their name - Vikings? :)

Monday, November 14, 2011

Creating a DSL for Cypher

With Cypher, querying data is like creating Ascii Art to navigate through information. While fun and powerful, sometimes the pesky execution engine doesn’t appreciate the beauty of what you’ve typed. In this blog post, Rickard Öberg presents his implementation of a DSL to safely construct Cypher queries in Java.

For example:
start( node( "n", 3, 1 ) ).
where( prop( "n.age" ).lt( 30 ).and( prop( "n.name" ).eq( "Tobias" )).
       or(not(prop("n.name").eq("Tobias" )))).
returns( nodes( "n" ) )

For more, read on at Rickard's blog.

Cheers,
Andreas

Thursday, November 10, 2011

Announcing Neo4j "Boden Bord" 1.5 GA Release

Hello graphistas!
After a successful Milestone 2 release of Neo4j 1.5 "Boden Bord" and excellent community and customer feedback, we've been busy at work putting the finishing touches to our Neo4j 1.5 GA release, which is now available on our downloads page. Since the last milestone you'll find we've smoothed a few rough edges and the documentation has been made really spick-and-span. We think this is our best release yet, but you can be the judge of that considering the splendid set of features and improvements that have gone into it.

Down in the engine room

The Neo4j kernel and other low-level components always gets some love from our hackers, and although not all of their efforts are immediately obvious as new shiny features their endeavours keep Neo4j getting smaller and faster (and brighter and shinier) every release. In the 1.5 GA release we've added some cool stuff and importantly we've taken some stuff out too — so your databases will be smaller on disk!
  • Property store(s) have a new layout on disk, resulting in roughly a 30% reduction in size and a speed bonus as fewer I/O read operations for reading properties. Correspondingly, we've updated the batch inserter and added a store migration utility that can be run on startup to migrate older stores.
  • There's a lot of useful system/environment logging that will appear in messages.log to help problem solving for lone hackers, community members, and our customers.

Server's up

The Neo4j server has received a lot of attention in this release, both in the Webadmin tool and the REST API.
Webadmin has been substantially refined and now offers style profiles that enable users to create rich, domain-specific graph visualizations. It also supports user-defined icons, which can really help with large/interconnected graphs. The visualization in Webadmin really looks super now, and so will your great domain-specific graphs. We'd love to see what you come up with.
For inspiration, take a look at what computational biologist @pablopareja has done with an excellent domain-specific visualization example in his post on microsatellites:

The Neo4j server has been upgraded to support:
  • HTTP request-level authorisation, fully customizable by developers.
  • More control over the server's IP address.
  • Lower memory overhead for the batch API and better response codes/hints when errors occur.
  • Simpler REST indexing APIs, particularly useful for folks on the .NET platform.

Cypher

The Cypher query language goes from strength to strength and is becoming an important pillar of how developers and users interact with graph data. Building on the solid foundation released in the previous milestone, the Cypher team have managed to move quickly enough to squeeze in even more features for this GA release. So in addition to all the goodies from previous releases:
  • Optional Relationships in match clauses.
  • New predicates ALL/ANY/NONE/SINGLE for iterables.
  • Path functions for NODES/RELATIONSHIPS/LENGTH.
  • Parameterizable literals,index queries and node/relationship id.
  • Support for the shortest path algorithm.
  • Bound relationships.
  • Sensible NULL checking.
  • A new aggregate function to COLLECT results.
Cypher now supports:
  • The DISTINCT modifier for all aggregate functions.
  • Nodes/relationships can be parameters.
And in a prolific display of developer documentation (!) productivity, our community lead Peter Neubauer has even started a Cypher cookbook to capture good idioms and Cypher patterns. It's also a great place to learn from and contribute to, so jump right in.

Highly Available, made Highly Awesome

Our team of tamed kernel hackers have been applying their talents to improving the operational characteristics of Neo4j's HA clustering. The HA protocol runs way faster now thanks to a revised asynchronous implementation, to the point where production clusters can be run spread across geographically remote locations on unpredictable cloud hardware. Talk about Web scale!
We've also made a set of changes under the covers to refine the way Neo4j clusters work at larger scales and distributions, including some level of self-healing when transaction managers restarted when they're found to be in a non-operational state, and proactive repairing of the underlying datastore after a full store copy from master so that log version is synced.
To make operating HA clusters a more convenient experience, we've added better support for keeping ZooKeeper under control and changed the names of some of the Neo4j config to de-emphasise ZooKeeper's role and keep cluster administration focussed on Neo4j (which means you might have to revise your deployment scripts to use Neo4j 1.5 GA).

Autumnal Blossoms

Following closely on the heels of the 1.5GA release, Spring Data Neo4j (SDN) will be going to 2.0.0.RELEASE. We've gotten great feedback from the SpringOne 2GX conference in Chicago, during which we released a milestone of SDN. Much code has been improved, examples polished and documentation updated with all the changes. If you're a Java enterprise developer, this is the hot package to warm up your code. Learn more by reading the guide book "Good Relationships" by our own Michael Hunger.

Get it while it's hot

We always value your feedback and would love for you to join our friendly and information-rich community mailing list.
The Neo4j 1.5 GA is ready for download, so download it and get involved
Until the next release, stay connected.
Jim Webber, Chief Scientist @ Neo Technology

Wednesday, November 9, 2011

Neo4j at the 1st London NOSQL Exchange!

Last week saw the first London NOSQL Exchange organised by Skillsmatter and Neo Technology. And what a success it was, with around 100 attendees and excellent talks throughout the day leading to excellent local beers throughout the evening (and the inevitable local hangover the following morning!).

Emil Eifrem in demand after his keynote

The Neo4j team was out in force, with Jim Webber (me!) chairing and occasionally refereeing the speakers. Following the chair's introduction, Neo Technology's CEO Emil Eifrem gave us a state of the NOSQL union talk where he looked into the past present, and future of NOSQL, guided by a graph traversal of course! Emil's talk reminded us of the impending "perfect wave" of data that is both voluminous and densely connected and highlighted where he thinks the various NOSQL database types are heading to try to accommodate this upheaval.

In one of the afternoon sessions, Neo Technology's Ian Robinson infected the audience with his passion for Doctor Who, and as a side-effect taught us all a great deal about Neo4j! Ian's talk took us through the history of TV's longest-running science-fiction show, and demonstrated how complex connected data can be managed within Neo4j, and importantly how business information (even geeky business information) can be mined from graph data using the cool Cypher query language. Wrapping this all up in mad cows and data provenance made for a hilarious and quirky look at how real world data concerns can be readily handled by Neo4j.

Later in the day Robert Rees of Wazoku gave an experience report on polyglot persistence using Neo4j and CouchDB. In the Wazoku use-case Neo4j is used to map associations between ideas while CouchDB is used to store disconnected documents with minimal supervision. While we think Neo4j is a great choice for storing documents too, it's fabulous to see that effective solutions can be built by combining NOSQL datastores in practice.

All in all the NOSQL Exchange was a superb event, and it was great to see so many old friends and meet new faces. We're hoping to do it again soon in London and perhaps even overseas too. So keep a lookout!