Java – Can’t connect to ElasticSearch server using Java API

elasticsearchjava

I am trying to connect to an ElasticSearch server using the Java API. I using elasticsearch service to start/stop and elasticsearch head for visualising the cluster. The cluster/node is active, the REST API works fine via curl on 9200. I have read pretty much every posts about this topic, yet I can't get it working, below are my details:

I have tried all these:

Using localhost,devhost1,127.0.0.1,or the actual IP on the network in TransportClient
Uncommenting the IP properties in config.yaml and putting 127.0.0.1 (server)
Checking if port 9300 is available, it is.

…9200/_cluster/nodes

{
  "ok": true,
  "cluster_name": "test",
  "nodes": {
    "NLVBbJpJTZWefeI2kQt3Tg": {
      "name": "inventory_management",
      "transport_address": "inet[/127.0.0.1:9300]",
      "hostname": "devhost1",
      "version": "0.90.5",
      "http_address": "inet[/127.0.0.1:9200]"
    }
  }
}

…9200/_cluster/health?pretty=true

{
  "cluster_name" : "test",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

Java Code:

Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "test").put("node.name", "inventory_management").build();
Client client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress("localhost", 9300));
IndexResponse response = client.prepareIndex("twitter", "tweet", "1").setSource(json).execute().actionGet();

Exception:

org.elasticsearch.transport.NodeDisconnectedException: [][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info]
org.elasticsearch.client.transport.NoNodeAvailableException: No node available
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:138)
    at org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java:124)
    at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java:242)
    at org.elasticsearch.client.action.index.IndexRequestBuilder.doExecute(IndexRequestBuilder.java:219)
    at org.elasticsearch.client.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:52)
    at org.elasticsearch.client.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47)

Log from Client (Java API):

[2013-10-24 16:37:15,783][DEBUG][threadpool.cached        ] [Aragorn] Initializing cached thread pool with keep_alive[1m], scheduled_size[20]
[2013-10-24 16:37:15,809][DEBUG][client.transport         ] [Aragorn] node_sampler_interval[1s]
[2013-10-24 16:37:15,820][DEBUG][netty.channel.socket.nio.NioProviderMetadata] Using the autodetected NIO constraint level: 0
[2013-10-24 16:37:15,872][DEBUG][transport.netty          ] [Aragorn] Connected to node [[#temp#-1][inet[localhost/127.0.0.1:9300]]]
[2013-10-24 16:37:15,892][DEBUG][transport.netty          ] [Aragorn] Disconnected from [[#temp#-1][inet[localhost/127.0.0.1:9300]]]
[2013-10-24 16:37:15,894][DEBUG][client.transport         ] [Aragorn] Failed to get node info from [#temp#-1][inet[localhost/127.0.0.1:9300]], removed from nodes list
org.elasticsearch.transport.NodeDisconnectedException: [][inet[localhost/127.0.0.1:9300]][/cluster/nodes/info]

+++++++++++++++++++++++

Tried with Unicast same error in the Java API.

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: "localhost" # Tried with devhost1 127.0.0.1 as well and ["localhost"] too

This is the server log:

[2013-10-25 13:49:58,379][INFO ][node                     ] [inventory_management] version[0.90.5], pid[2426], build[c8714e8/2013-09-17T12:50:20Z]
[2013-10-25 13:49:58,379][INFO ][node                     ] [inventory_management] initializing ...
[2013-10-25 13:49:58,382][INFO ][plugins                  ] [inventory_management] loaded [], sites []
[2013-10-25 13:49:59,853][INFO ][node                     ] [inventory_management] initialized
[2013-10-25 13:49:59,853][INFO ][node                     ] [inventory_management] starting ...
[2013-10-25 13:49:59,937][INFO ][transport                ] [inventory_management] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.1.50:9300]}
[2013-10-25 13:50:02,960][INFO ][cluster.service          ] [inventory_management] new_master [inventory_management][2-uG1xVNSSiLo5RVRrjbGg][inet[/192.168.1.50:9300]], reason: zen-disco-join (elected_as_master)
[2013-10-25 13:50:02,972][INFO ][discovery                ] [inventory_management] test/2-uG1xVNSSiLo5RVRrjbGg
[2013-10-25 13:50:02,989][INFO ][http                     ] [inventory_management] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.1.50:9200]}
[2013-10-25 13:50:02,989][INFO ][node                     ] [inventory_management] started
[2013-10-25 13:50:03,024][INFO ][gateway                  ] [inventory_management] recovered [0] indices into cluster_state
[2013-10-25 13:50:09,399][WARN ][transport.netty          ] [inventory_management] exception caught on transport layer [[id: 0x02649775, /127.0.0.1:50028 => /127.0.0.1:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format
    at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
[2013-10-25 13:50:09,403][WARN ][transport.netty          ] [inventory_management] exception caught on transport layer [[id: 0x02649775, /127.0.0.1:50028 :> /127.0.0.1:9300]], closing connection
java.io.StreamCorruptedException: invalid internal transport message format
    at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:482)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:361)
    at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
    at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
    at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
    at org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
    at org.elasticsearch.transport.netty.NettyTransport.exceptionCaught(NettyTransport.java:501)
    at org.elasticsearch.transport.netty.MessageChannelHandler.exceptionCaught(MessageChannelHandler.java:228)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
    at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

+++++++++++++++++++++++
+++++++++++++++++++++++

Java version:

java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.12) (7u25-2.3.12-4ubuntu3)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

+++++++++++++++++++++++
+++++++++++++++++++++++
+++++++++++++++++++++++

Issue has been solved as per below. Make sure your client and server versions are in sync!

Best Answer

Just so this question is marked answered for others who may experience the same issue (the answer is actually at the end of the question).

I ran into the same issue and it turned out to be a discrepancy between the version of the JAR used by the Java client and the version the server was running. Your best bet is to ensure an exact match, then the instructions given here just work w/o any tweaking.

Another thing to check is that your Java client is using the correct port, which is not the one used by http clients. The correct port defaults to 9300 rather than 9200, which is used by the latter.

Related Solutions

Java – How to avoid Java code in JSP files, using JSP 2

The use of scriptlets (those <% %> things) in JSP is indeed highly discouraged since the birth of taglibs (like JSTL) and EL (Expression Language, those ${} things) way back in 2001.

The major disadvantages of scriptlets are:

Reusability: you can't reuse scriptlets.
Replaceability: you can't make scriptlets abstract.
OO-ability: you can't make use of inheritance/composition.
Debuggability: if scriptlet throws an exception halfway, all you get is a blank page.
Testability: scriptlets are not unit-testable.
Maintainability: per saldo more time is needed to maintain mingled/cluttered/duplicated code logic.

~~Sun~~ Oracle itself also recommends in the JSP coding conventions to avoid use of scriptlets whenever the same functionality is possible by (tag) classes. Here are several cites of relevance:

From JSP 1.2 Specification, it is highly recommended that the JSP Standard Tag Library (JSTL) be used in your web application to help reduce the need for JSP scriptlets in your pages. Pages that use JSTL are, in general, easier to read and maintain.

...

Where possible, avoid JSP scriptlets whenever tag libraries provide equivalent functionality. This makes pages easier to read and maintain, helps to separate business logic from presentation logic, and will make your pages easier to evolve into JSP 2.0-style pages (JSP 2.0 Specification supports but de-emphasizes the use of scriptlets).

...

In the spirit of adopting the model-view-controller (MVC) design pattern to reduce coupling between the presentation tier from the business logic, JSP scriptlets should not be used for writing business logic. Rather, JSP scriptlets are used if necessary to transform data (also called "value objects") returned from processing the client's requests into a proper client-ready format. Even then, this would be better done with a front controller servlet or a custom tag.

How to replace scriptlets entirely depends on the sole purpose of the code/logic. More than often this code is to be placed in a fullworthy Java class:

If you want to invoke the same Java code on every request, less-or-more regardless of the requested page, e.g. checking if a user is logged in, then implement a filter and write code accordingly in doFilter() method. E.g.:

  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws ServletException, IOException {
      if (((HttpServletRequest) request).getSession().getAttribute("user") == null) {
          ((HttpServletResponse) response).sendRedirect("login"); // Not logged in, redirect to login page.
      } else {
          chain.doFilter(request, response); // Logged in, just continue request.
      }
  }

When mapped on an appropriate <url-pattern> covering the JSP pages of interest, then you don't need to copypaste the same piece of code overall JSP pages.

If you want to invoke some Java code to process a GET request, e.g. preloading some list from a database to display in some table, if necessary based on some query parameters, then implement a servlet and write code accordingly in doGet() method. E.g.:
```
  protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      try {
          List<Product> products = productService.list(); // Obtain all products.
          request.setAttribute("products", products); // Store products in request scope.
          request.getRequestDispatcher("/WEB-INF/products.jsp").forward(request, response); // Forward to JSP page to display them in a HTML table.
      } catch (SQLException e) {
          throw new ServletException("Retrieving products failed!", e);
      }
  }
```
This way dealing with exceptions is easier. The DB is not accessed in the midst of JSP rendering, but far before the JSP is been displayed. You still have the possibility to change the response whenever the DB access throws an exception. In the above example, the default error 500 page will be displayed which you can anyway customize by an <error-page> in web.xml.

If you want to invoke some Java code to process a POST request, such as gathering data from a submitted HTML form and doing some business stuff with it (conversion, validation, saving in DB, etcetera), then implement a servlet and write code accordingly in doPost() method. E.g.:

  protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      String username = request.getParameter("username");
      String password = request.getParameter("password");
      User user = userService.find(username, password);

      if (user != null) {
          request.getSession().setAttribute("user", user); // Login user.
          response.sendRedirect("home"); // Redirect to home page.
      } else {
          request.setAttribute("message", "Unknown username/password. Please retry."); // Store error message in request scope.
          request.getRequestDispatcher("/WEB-INF/login.jsp").forward(request, response); // Forward to JSP page to redisplay login form with error.
      }
  }

This way dealing with different result page destinations is easier: redisplaying the form with validation errors in case of an error (in this particular example you can redisplay it using ${message} in EL), or just taking to the desired target page in case of success.

If you want to invoke some Java code to control the execution plan and/or the destination of the request and the response, then implement a servlet according to the MVC's Front Controller Pattern. E.g.:

  protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      try {
          Action action = ActionFactory.getAction(request);
          String view = action.execute(request, response);

          if (view.equals(request.getPathInfo().substring(1)) {
              request.getRequestDispatcher("/WEB-INF/" + view + ".jsp").forward(request, response);
          } else {
              response.sendRedirect(view);
          }
      } catch (Exception e) {
          throw new ServletException("Executing action failed.", e);
      }
  }

Or just adopt an MVC framework like JSF, Spring MVC, Wicket, etc so that you end up with just a JSP/Facelets page and a JavaBean class without the need for a custom servlet.

If you want to invoke some Java code to control the flow inside a JSP page, then you need to grab an (existing) flow control taglib like JSTL core. E.g. displaying List<Product> in a table:
```
  <%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
  ...
  <table>
      <c:forEach items="${products}" var="product">
          <tr>
              <td>${product.name}</td>
              <td>${product.description}</td>
              <td>${product.price}</td>
          </tr>
      </c:forEach>
  </table>
```
With XML-style tags which fit nicely among all that HTML, the code is better readable (and thus better maintainable) than a bunch of scriptlets with various opening and closing braces ("Where the heck does this closing brace belong to?"). An easy aid is to configure your web application to throw an exception whenever scriptlets are still been used by adding the following piece to web.xml:
```
  <jsp-config>
      <jsp-property-group>
          <url-pattern>*.jsp</url-pattern>
          <scripting-invalid>true</scripting-invalid>
      </jsp-property-group>
  </jsp-config>
```
In Facelets, the successor of JSP, which is part of the Java EE provided MVC framework JSF, it is already not possible to use scriptlets. This way you're automatically forced to do things "the right way".
If you want to invoke some Java code to access and display "backend" data inside a JSP page, then you need to use EL (Expression Language), those ${} things. E.g. redisplaying submitted input values:
```
  <input type="text" name="foo" value="${param.foo}" />
```
The ${param.foo} displays the outcome of request.getParameter("foo").
If you want to invoke some utility Java code directly in the JSP page (typically public static methods), then you need to define them as EL functions. There's a standard functions taglib in JSTL, but you can also easily create functions yourself. Here's an example how JSTL fn:escapeXml is useful to prevent XSS attacks.
```
  <%@ taglib uri="http://java.sun.com/jsp/jstl/functions" prefix="fn" %>
  ...
  <input type="text" name="foo" value="${fn:escapeXml(param.foo)}" />
```
Note that the XSS sensitivity is in no way specifically related to Java/JSP/JSTL/EL/whatever, this problem needs to be taken into account in every web application you develop. The problem of scriptlets is that it provides no way of builtin preventions, at least not using the standard Java API. JSP's successor Facelets has already implicit HTML escaping, so you don't need to worry about XSS holes in Facelets.

Best Answer

Related Solutions

Java – How to avoid Java code in JSP files, using JSP 2

See also:

Java – elasticsearch RemoteTransportException on dedicated server

Related Topic