Best Way to Store and Load a Large Word List in Java

I use the Externalizable interface to solve this problem. It doesn't really meet your Appendable requirement, but it might get you started.

Externalizable lets you write each object out yourself. What I do is include a version number for each class. Then, when I change the class, I adjust the readExternal method so it can can read the new format as well as the old ones and change the writeExternal method to write the new format with the new version number. The code to read old formats may need to change to fill new fields (and not fill in removed fields). Then I'm ready to go.

There's usually some sort of object hierarchy in the graph, so if a level 2 class is seriously modified, the new write/read for that class can output/input something completely different from the old one, with completely new lower-level classes.

Some care is needed because sometimes there isn't a hierarchy. You have to remember that, after reading, you can't set a field by pulling data from objects in fields because they may not really be there yet. (Sometimes I add a setUp method to all my classes that gets called after the top-level objectRead so the objects in the graph can get themselves organized after the whole graph has been read.

And it sometimes helps to write out objects manually using a method that just writes out plain bytes. (But on the read you have to know precisely what you are reading, where readExternal will read a complete object of any class.)

Code looks something like:

// version 3 -- November 16, 2013
// version 2 -- March 22, 2013
// version 1 -- April 1, 2012
public void writeExternal( ObjectOutput out )  throws IOException  {
    out.writeShort( 3 )
    out.writeLong( longData );
    out.writeObject( something );
    ....
}

public void readExternal( ObjectInput in )  throws IOException,
             ClassNotFoundException  {
    short version = in.readShort();
    if (version > 3)  {
        // Admit program is too old to read file.
    }
    else if (version == 3)  {
        longData = in.readLong();
        something = readobject();
    }
    else if (version == 2)  {
        longData = 5;
        something = readobject();
    }
    else
       ...
}

The easiest approach if you can accept the drawbacks is to have tomcat as a passive server such that it never contacts the server app. All data flows the other way by periodicly sending heart beats to tomcat and retrieve the json data you mentioned. The drawbacks is that it will (only) be eventual consistent so you will have to accept a small time delay.

The other approach is hosting a server in the server app. This can be a RMI server or a http server. Since java SE 6 there have been a http server built into java which is fairly easy to work for posting documents back and forth. There is no templating or all that stuff, but just serialize your data to JSON and you are good to go. Once you have this server running just connect to it from the tomcat server.