Datomic Using Postgres Back-End

This started out as a post to the datomic mailing list, asking for help.

I try to ask not-stupid questions, so I kept picking at the obvious “what happens if you do x?” style questions until I wound up arriving at an actual answer.

Background

This may not even be a supported configuration, but it seemed like it might be worth mentioning, if only for its “that seems odd” factor. If nothing else, maybe it’ll help anyone else who runs into this and tries to google for it.

I recently set up datomic pro starter using postgres as the back-end on ubuntu 14.04, following the instructions at http://docs.datomic.com/storage.html. I don’t remember having any issues at all.

A couple of weeks ago, I set up a vagrant box based on the latest 14.04, and have had some issues getting datomic working with postgres 9.4.

First Strangeness

This time around, I couldn’t just run datomic’s bin/sql scripts specifying -U postgres. I had to actually sudo over to that user first.

Transactor Failure

After that was configured, the initial output from starting the transactor (I haven’t changed anything in the default sql config file except adding my license key) looked normal at first:

Launching with Java options -server -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50
Starting datomic:sql://<DB-NAME>?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic, you may need to change the user and password parameters to work with your jdbc driver ...
System started datomic:sql://<DB-NAME>?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic, you may need to change the user and password parameters to work with your jdbc driver

After about a minute, it exited with:

Critical failure, cannot continue: Lifecycle thread failed
java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: ERROR: relation "datomic_kvs" does not exist
  Position: 31
       at java.util.concurrent.FutureTask.report(FutureTask.java:122)
       at java.util.concurrent.FutureTask.get(FutureTask.java:188)
       at clojure.core$deref_future.invoke(core.clj:2180)
       at clojure.core$future_call$reify__6320.deref(core.clj:6420)
       at clojure.core$deref.invoke(core.clj:2200)
       at datomic.lifecycle_ext$standby_loop.invoke(lifecycle_ext.clj:28)
... (my apologies for not including the entire stack trace. I haven't managed to get copy/paste working out of this VM yet)

I’m not all that familiar with postgres. It took me a while to figure out that the trick to connecting to the datomic database as the datomic role via the command line is to run:

psql datomic -h 127.0.0.1 -d datomic

This is really the first time I’ve used the command-line client. The part that caught my eye was that, as the postgres user, running \dt showed something like

                    List of relations
  Schema |      Name       |  Type  |  Owner
-------------+------------------+---------+----------
   public    | datomic_kvs | table    | postgres

Running that same command as the datomic user printed

"No relations found."

My current guess about what went wrong is that I had a typo when I ran the postgres-table.sql script, and it created the datomic_kvs table in the postgres database.

I ran that “create table …” command manually, and the transactor seems to be running without a hitch.

I don’t know whether there might be any impact from using the 9.3 drivers against a 9.4 install. It seems dubious, but I just don’t know enough about postgres to even start trying to hazard a guess on this one.

Client Side

After I wrote that and got a good night’s sleep, I tackled the interesting part: the client.

Trying to create a new database results in:

HornetQConnectionTimedOutException HQ119013: Timed out waiting to receive cluster topology. Group:nell org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory (ServerLocatorImpl.java:946)

I switched ports in the connection string, just for grins to see what would happen. That changed the problem to an EOFException.

So I tried out the basic commands from the “getting started guide” using the groovysh script that comes with datomic. That part worked fine.

When I update the connection string to the database that I created that way and just try to connect through a clojure REPL, the error switches to

CompilerException clojure.lang.lang.ExceptionInfo: Error communicating with HOST localhost on PORT 4334 {:alt-host nil, :peer-version 2, :password "...", :username "...", :port 4334, :host "localhost", :version "0.9.5153", :timestamp 1427687038138, :encrypt-channel true}, compiling:(form-init1256469964991696942.clj:1:17)

It looks like that came from nrepl middleware. It’s

Caused by:
ExceptionInfo: Error communicating with HOST localhost on PORT 4334 {:alt-host nil, :peer-version 2, :password "...", :username "...", :port 4334, :host "localhost", :version "0.9.5153", :timestamp 1427687038138, :encrypt-channel true}
  clojure.core/ex-info (core.clj:4577)
  datomic.connector/endpoint-error (connector.clj:52)
  datomic.connector/create-hornet-factory (connector.clj:148)
  datomic.connector/create-transactor-hornet-connector (connector.clj:327)
  datomic.connector/create-transactor-hornet-connector (connector.clj:325)
  datomic.peer.Connection/fn--8924 (peer.clj:235)
  datomic.peer.Connection (peer.clj:,221)
  datomic.peer/create-connection/reconnect-fn--8985 (peer.clj:488)
  datomic.peer/create-connection (peer.clj:498)
  datomic.peer/get-connection/fn--9039/fn--9041 (peer.clj:665)
  datomic.peer/get-connection/fn--9039 (peer.clj:662)
  datomic.peer/get-connection (peer.clj:656)
Caused by:
HornetQConnectionTimedOutException HQ19013: Timed out waiting to receive cluster topology. Group:null
  org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory (ServerLocatorImpl.java:946)
  datomic.hornet/create-session-factory (hornet.clj:220)
  clojure.core/apply (core.clj:628)
  datomic.connector/try-hornet-connect (connector.clj:108)
  datomic.connector/create-hornet-factory (connector.clj:143)

There are a handful of google results about that final issue, but they all seem to ultimately wind up with the author answering “Oh, I see what I did wrong.

Which leaves me hopeful, but isn’t very helpful.

I headed back to the project roots. I figured it would be simpler to just create a new project, verify that it can connect with nothing but the bare bones, and then add in one piece at a time to see where it starts failing.

The very first difference that stood out was that, without doing anything special, datomic pro’s hornetq-server “2.3.17.Final” overrides immutant’s “2.4.5.Final”. And then that leads to datomic’s hornetq-journal “2.3.17.Final” overriding immutant’s hornetq-journal “2.4.5.Final”. And there are a slew of others along the same lines.

My inclination here is to update immutant, but they’re already using newer libraries.

Instead, I trimmed back that project definition so the datomic-pro dependency got rid of its hornetq-server exclusion. At first glance, that seems to have done the trick.

My initial guess is that the root cause is that hornetq 2.4.x won’t communicate with 2.3.x, but that seems like a really bad guess. I know even less about hornetq than I do about postgres, so that may not even be in the right ballpark. If I had more time to devote to this sort of thing, I’d try to track this down to get a definite answer. As things stand, I’m happy to have an answer that’s good enough to let me move on.