Comments on Neo4j Blog: The current database debate and graph databases

ªAnonymous: At the time Scott Wheeler tried out Ne...

2010-03-12T14:22:24.664+01:00

ªAnonymous: At the time Scott Wheeler tried out Neo4j, it didn't have a batch inserter, so inserting large datasets wasn't nearly as fast as it is now. I guess this solves at least part of their problem. As we never got any detailed information regarding the problem, it's hard to be more specific.

im interested to hear if Scott Wheeler (of Directe...

2010-03-04T08:54:57.016+01:00

im interested to hear if Scott Wheeler (of DirectedEdge) problems with Neo4j performance have been solved?

At least some of the things that Phil Goetz misses...

2009-09-10T12:28:42.373+02:00

At least some of the things that Phil Goetz misses in Neo4J are, as far as I can see, to be found in Topic Maps. Check out e.g. the implementation by Ontopia http://code.google.com/p/ontopia/. The query language that is being developed for Topc Maps is also based on Prolog.

While you are correct in your assesment that key v...

2009-07-31T23:29:43.419+02:00

While you are correct in your assesment that key value stores dont have any relationships between the key value pairs. But you should also mention that the main draw to key value stores is their distributability. I just started reading about Neo4j so I dont know how ready it is to run on data spread over 100s of different machines or to be run in an online environment.

@Phil Goetz Thanks for your input! Neo4j was des...

2009-07-21T23:17:49.958+02:00

@Phil Goetz

Thanks for your input!

Neo4j was designed to provide a simple, flexible and performant way to persist directed labeled graphs. Other representations can then easily be built on top of Neo4j and as different use cases have different needs, it wouldn't be a good design choice to try to put everything into the core Neo4j engine.

At the moment Neo4j is quite new and doesn't come with lots of different layers built on top of it, but this part improves more and more. One example of a layer on top of Neo4j that already exists is the RDF/SPARQL implementation, maybe that is of some interest for you.

To solve your problems, you could for example add a layer to represent rules using Neo4j. Also, you could add a layer to make relationships able to have relationships themselves by simply using an intermediate node. I'm sure you could get some help from people on the mailing list.

I was excited to hear about Neo4j. I was disappoi...

2009-07-21T02:23:01.738+02:00

I was excited to hear about Neo4j. I was disappointed to see how it's implemented.

It's a vast improvement over relational DBs. You should write examples for the SQL people who can't see that. For example, here's an SQL query in a program I work on:

SELECT r.role_id FROM omnium..ident_names i, omnium..role_link r, omnium..asm_feature f, omnium..db_data d WHERE i.lower_com_name = ? AND i.locus = r.locus AND i.locus=f.locus AND f.db_data_id=d.id AND (d.original_db < 'nt' OR d.original_db >= 'nu')

With a logic system, the query would be

com_name(L, ?), original_db(L, DB), or(less(DB, 'nt'), less('nu', DB)), role_id(L, R).

The query is simpler because for each node, every fact about that node is connected to it, so you don't have to know what table the info is in, or query fields like db_data_id whose only purpose is to link to tables together.

Also, with a logic system, you can add rules that will then be automatically used to answer queries. So if you have to look up role_id by com_name a lot, you add the rule

comname_roleid(CN, R) :- com_name(L, CN), original_db(L, DB), or(less(DB, 'nt'), less('nu', DB)), role_id(L, R).

and then can just ask

comname_roleid('ATP synthase', ?)

You can put more and more intelligence into the graph itself.

More importantly, using a relational DB hinders you from using the information you have. For instance, if you have a reaction pathway database, it would be really hard to write an SQL query that meant 'Find all the proteins X whose mutation could account for the loss of production of protein Y." It would be simple with a graph database.

Sadly, you can't do all of these things with neo4j either! It falls short of the basic principles of knowledge representation developed in the 1970s in 3 critical ways:

1. It stores only triples. It can't represent unary or trinary relations. This means that it can't represent the "not" predicate. So if your schema lets you store "relation(A,B)" in the DB, you can't store the fact "not(relation(A,B))". Also, you can't store the relation "true(relation(A,B))", which means you can't distinguish between facts that are asserted as true, and facts that are merely the constituents of larger, opposite assertions, like "not(relation(A,B))". This means you can't do reasoning, because you can't represent what one agent believes that another agent believes.

2. The 'link' part of a neo4j relation is not a node! So you can't write a relation about another relation. So you can't even represent boolean statements like and(or(A,B), or(C,D)). All the info about a link has to be stored in that link, so it is impossible to add arbitrary modifiers about links! For instance, if you have a link saying that a gene has an annotation, you can't then say how confident you are about that annotation, except by building that into the link schema. And even if you do that, you can't then make another assertion about your confidence in the data you used to make the first confidence assertion. (Yes, it is necessary to do this in the real world.) Which defeats the purpose of moving away from a relational DB.

3. It doesn't use the same representation for rules and data. So there are a lot of things you can't do. You can't use machine learning to learn rules; only programmers can write rules, because they are in Java code, not in Java data structures. You can't write rules to inspect other rules; for example, to prioritize rules based on their complexity.

What I want is a Prolog that works using terabyte knowledge stores on disk. See SNePS for an example of a system that gets knowledge representation right. Unfortunately, I think it requires its knowledge to all be in memory.

One of the other big issues is that a lot of datab...

2009-04-08T23:22:23.450+02:00

One of the other big issues is that a lot of database work ends up as follows...

1) Using JPA (or whatever), serialize and de-serialize a lot of object graphs. This happens a lot interactively.

2) Occasionally (less often than (1) but still fairly often), create broad-based sweeping queries against the data represented by those object graphs.

RDBMS systems can handle (1) moderately poorly, and handle (2) really, really well. Graph and Object systems handle (1) well and (2) poorly. The thing is that, generally, I'd almost always take a trade-off if the interactive graph building was 10x slower and the big, hairy, complicated reporting was 10x faster, based on application usage.

Every now and then I start feeling differently, and make a system that's more optimized for OO usage than for data usage. Every single time, so far, I've come to regret that decision.

"People are questioning the RDBMS to an extent tha...

2009-04-06T16:19:04.553+02:00

"People are questioning the RDBMS to an extent that we're not used to."

I commented on that article on my blog, so I won't repeat my arguments here. After debating with a commenter, I'm inclined to think that this "new energy" is another round of inventors trying to attract attention to their new DBMS. I don't think there's anything wrong with that; after all, if they really do have a good idea it ought to get attention. I just think there's no reason we should be less skeptical of hype or FUD than we normally are.

Ðis is ſtupid. People condemn ðe relational model...

2009-04-06T15:29:25.722+02:00

Ðis is ſtupid. People condemn ðe relational model for SQL limitations, but SQL is not relational at all. Much better ðan reinventing ðe wheel would be to implement truly relational databaſe management ſyſtems.

There is one novel and rather unusual approach whe...

2009-04-06T14:01:38.306+02:00

There is one novel and rather unusual approach where partially ordered sets are used instead of graphs: http://c2.com/cgi/wiki?ConceptOrientedModel What is attractive is that it is claimed to be joinless and integrated with programming (like LINQ?). Yet, it is not clear if it is something really new or simply a number of old techniques and patterns.