The ElasticSearch Java API

When you start diving into the ElasticSearch Java API, you will undoubtedly note that it is kind of a mystery. There is the official documentation and there is that what’s missing in these pages.

ElasticSearch is an amazing product, but using it in Java requires patience for searching in blogs, Stack Overflow posts and even the source code itself. Diving into the source code is actually an excellent way of getting to understand the Java API. The source code – implementations and tests – are a great place to find examples and use cases. However, to quote Simon Brown on this: “The code doesn’t tell the whole story“.

In this article, I’ll start with some code snippets, a kind of “How to” collection. Most of these informations are coming from somewhere on the internet (thank you fellow bloggers and SO problem solvers :) or from the source codes.

Let’s start with the Maven pom file that I used to compile these examples. The “exec-maven-plugin” allows to start the application via Maven (if you want to do this).

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>pro.carl</groupId>
    <artifactId>estest</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>estest</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.1.3</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <configuration>
                    <mainClass>pro.carl.estest.App</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

The first thing to do in a Java ElasticSearch client application is to create a “client” (API access) object. There are two types of “client” implementations: the Node Client and the Transport Client. This is well described in the official documentation, so I’ll not discuss this here.

ElasticSearch uses the Builder patter for everything. To build the object required in order to establish a connection to the ElasticSearch cluster, we use the “NodeBuilder“. Again, the source files are the source for a great deal of information.

Node node = null;
Client client = null;
try {

    // Instantiates an ElasticSearch cluster node in the
    // current VM. The behavior of this node is defined by the
    // settings we specify. "client(true)" indicates that we
    // going to be a pure client node, which means it will
    // hold no index data and other optimizations are applied
    // by different modules.
    // The node tries to find a master node to connect too and
    // has a default timeout of 30 seconds.
    node = NodeBuilder.nodeBuilder().client(true).node();

    // This is actually the object that allows us to execute
    // commands against our local node and by this the entire
    // cluster.
    client = node.client();

    :
    // check if all is GREEN and do your work here...
    :
 }
 finally {
    if (node != null) node.close();
 }

Normally, the node configuration is loaded from the “elasticsearch.yml” file in your classpath, but everything can be setup programmatically as well.

Setting the cluster name:

node = NodeBuilder.nodeBuilder().
                   client(true).
                   clusterName("mycluster").
                   node();

If you have a look into the source code, you will see that some of these methods are just adding values to the internal settings properties file – no magic here. Consequently, we can configure our cluster in the same way.

Settings settings = ImmutableSettings.settingsBuilder().
                           put("cluster.name", "myclustername").
                           put("node.data", false).
                           put("node.name", "mynodename").
                           build();

node = NodeBuilder.nodeBuilder().
                           settings(settings).
                           node();
client = node.client();

If you are curious to see what you are getting back, hidden behind the “Node” interface, again let’s look into the source code. Note all the code around the thread pool handling…

Following this, you need to check the cluster status before doing the real work. The cluster status has 3 possible values: RED, YELLOW and GREEN. Consult the documentation to get more information.

ClusterHealthResponse hr=null;
try {
    hr=client.admin().cluster().
                 prepareHealth().
                 setWaitForGreenStatus().
                 setTimeout(TimeValue.timeValueMillis(250)).
                 execute().
                 actionGet();
}
catch(MasterNotDiscoveredException e) {
    // No cluster status since we don't have a cluster
}

if (hr!=null) {
    System.out.println("Data nodes found:"
                   +hr.getNumberOfDataNodes());
    System.out.println("Timeout? :"
                   +hr.isTimedOut());
    System.out.println("Status:"
                   +hr.getStatus().name());
}

One additional comment on the cluster status: if you specify the requirement of one replica at index creation, but you only have one single data node running. Your status will never pass to GREEN, but will remain YELLOW – the requested replica is not present. This may happen during development where typically one single data node is running in a corner. In a development environment, this may be ignored. In a production environment on the other hand, this indicates that one of the nodes is not working as expected and no other data was able to take over!

Relates source codes: ClusterHealthResponse, and documentation on Cluster Health.

An important note concerning the “actionGet()“: The name of this method is not related to a potential GET REST call to the cluster. In fact, the “execute()” method returns a “ListenableActionFuture” object, which is the descriptor of the operation that is executed asynchronously in a separate thread. In order to wait for the result and making the operation a blocking call, the “afterGet()” method must be invoked on this Future.

Before we continue with handling indices, types and mappings, you should get acquainted with these concepts, if not yet done: Basic ElasticSearch Concepts.

The next step would be creating an index:

CreateIndexRequestBuilder cirb = client.admin().
                                    indices().
                                    prepareCreate("myindexname");

HashMap<String,Object> settings = new HashMap<>();

// Choosing the number of shards is actually an important one.
// This is well documented in the official documentation.
// The default value is 5, but we keep all data in 1 single shard
settings.put("number_of_shards", 1);

// Number of replication of indices within the cluster
settings.put("number_of_replicas", 1);

cirb.setSettings(settings);

CreateIndexResponse createIndexResponse=null;
try {
    createIndexResponse = cirb.execute().actionGet();
}
catch(IndexAlreadyExistsException e) {
    // Index already exists
    return;
}

if (createIndexResponse!=null && 
     createIndexResponse.isAcknowledged()) {
    // Index created
    return;
}
else {
    // Index creation failed
    return;
}

Once the index is created, we add a mapping associated with an index type. ElasticSearch can work in a schema-less way, but it’s generally a good practice to use a well defined schema.

ElasticSearch helps with this be allowing to set the “dynamic” schema mode to “strict”. As consequence, ElasticSearch throws an exception when you try to populate the index with data that does not respect the schema i.e. the mapping. This setting is off by default, which signifies that ElasticSearch can add fields on the fly to the mapping when they appear in the data set. See the official documentation for more info on the dynamic mapping settings.

The creation of a “new” mapping over an existing mapping merges these mappings and an error will be throw when incompatible types are detected. To avoid trouble, it’s a good practice to check if a mapping exists before creating a new (different) one. After merging mappings, it’s likely that the index has to be rebuild to re-populate the new fields or update the existing ones in case the analyser rules have changed.

There are multiple possibilities to create a type/mapping. We can use a classic JSON file or the ElasticSearch builder.

XContentBuilder builder = XContentFactory.jsonBuilder().
          startObject().
             startObject(TYPE).
                field("dynamic", "strict").
                startObject("_id").
                     field("path", "id").
                endObject().
                startObject("_all").
                     field("enabled", "true").
                endObject().
                startObject("properties").
                     startObject("id").
                         field("type", "long").
                         field("store", "yes").
                         field("index", "not_analyzed").
                     endObject().
                     startObject("country_code").
                         field("type", "string").
                         field("store", "yes").
                         field("index", "not_analyzed").
                     endObject().
                     startObject("names").
                         field("type", "string"). // (*)
                         field("store", "yes").
                         field("index", "analyzed").
                     endObject().
                     startObject("postal_codes").
                         field("type", "string").
                         field("store", "yes"). 
                         field("index", "analyzed").
                     endObject().
                endObject().
            endObject().
        endObject();

PutMappingResponse response=client.admin().
                           indices().
                           preparePutMapping("myindexname").
                           setType("mytypename").
                           setSource(builder).
                           execute().
                           actionGet();

if (response.isAcknowledged()) {
    // Type and Mapping created!
}
else {
    // Failed to create type and mapping
}

(*) A good thing to know about the “type” in mappings is that ElasticSearch treats them as arrays of the specified type. This means that the field “names” of type “string” can actually contain [“name1″, “name2″, “name3″] – or just “name”.

Note the “field(“dynamic”, “strict”)” part of the mapping (see above).

It’s also possible to have more complex structures with nested objects. In this example, the names are nested inside the postal code.

XContentBuilder builder = XContentFactory.jsonBuilder().
    startObject().
       startObject(TYPE).
          field("dynamic", "strict").
          startObject("_id").
             field("path", "id").
          endObject().
          startObject("_all").
             field("enabled", "true").
          endObject().
          startObject("properties").
              startObject("id").
                 field("type", "long").
                 field("store", "yes").
                 field("index", "not_analyzed").
              endObject().
              startObject("country_code").
                 field("type", "string").
                 field("store", "yes").
                 field("index", "not_analyzed").
              endObject().
              startObject("postal_codes").
                 field("type", "nested").
                 startObject("properties").
                     startObject("code").
                         field("type", "string").
                         field("store", "yes").
                         field("index", "analyzed").
                     endObject().
                     startObject("names").
                         field("type", "nested").
                         startObject("properties").
                             startObject("name").
                                 field("type", "string").
                                 field("store", "yes").
                                 field("index", "analyzed").
                             endObject().
                         endObject().
                     endObject().
                 endObject().
              endObject().
         endObject().
    endObject();

I would suggest to apply the KISS rule in this case ;) The “XContentBuilder” has also a very handy “toString()” method.

You can of course use plain old JSON as input. This comes very handy when the schema is read from application settings, which would allow to play with different analysers, for example, without redeploying the entire application. A re-indexing however is always required.

String mappingString="{"\"dynamic\":\"strict\",\"_id\":{\"path\":\"id\"},\"properties\":{\"country_code\":{\"type\":\"string\",\"index\":\"not_analyzed\",\"store\":true},\"id\":{\"type\":\"long\",\"store\":true},\"names\":{\"type\":\"string\",\"store\":true},\"postal_codes\":{\"type\":\"string\",\"store\":true}}}";

PutMappingResponse response=client.admin().
                              indices().
                              preparePutMapping("myindexname").
                              setType("mytypename").
                              setSource(mappingString).
                              execute().
                              actionGet();

Now, let’s see different operations that can be executed on the index:

Read the mapping of an index and type:

IndexMetaData imd = null;
try {
    ClusterState cs = client.admin().
                         cluster().
                         prepareState().
                         setIndices("myindexname").
                         execute().
                         actionGet().
                         getState();
   
    imd = cs.getMetaData().index("myindexname");
}
catch (IndexMissingException e) {
    // If there is no index, there is no mapping either
}

MappingMetaData mdd = imd.mapping(type);

if (mdd == null) {
    // No mapping found
}
else {
    System.out.println("Mapping as JSON string:" + mdd.source());
}

This verifies if a type exists on an index. It does however NOT verify if the type has a mapping defined.

client.admin().indices().
               prepareTypesExists("myindexname").
               setTypes("mytypename").
               execute().
               actionGet().
               isExists();

The next command verifies if an index exists. It does not verify if there is data in the index.

client.admin().indices().
               prepareExists("myindexname").
               execute().
               actionGet().
               isExists();

Now, we verify if a mapping exists on an index and type:

IndexMetaData imd = null;
try {
    ClusterState cs = client.admin().
                             cluster().
                             prepareState().
                             setIndices("myindexname").
                             execute().
                             actionGet().
                             getState();

    imd = cs.getMetaData().index(index);
}
catch (IndexMissingException e) {
   // If there is no index, there is no mapping either
   return false;
}

MappingMetaData mdd = imd.mapping(type);

if (mdd != null)
    return true;

return false;

Deleting a mapping is simple:

client.admin().indices().
               prepareDeleteMapping("myindexname").
               setType(type).
               execute().
               actionGet();

And deleting an index too:

DeleteIndexResponse rep = null;
try {
    rep = client.admin().
                 indices().
                 prepareDelete("myindexname").
                 execute().
                 actionGet();
}
catch (IndexMissingException e) {
    // Index not found
    return;
}

if (rep.isAcknowledged()) {
    // Index deleted
}
else {
    // Failed to delete index
}

An alias is a logical name that can be assigned to an index and then used instead of the index name to execute commands against that index. Aliases are very helpful when you need to seamlessly rebuild an index.

To add an alias, do…

try {
    client.admin().
           indices().
           prepareAliases().
           addAlias("myindexname", "myaliasname").
           execute().
           actionGet();
}
catch(IndexMissingException e) {
    // Index not found
}

Switching an alias over from one index to another index, in one single atomic operation, can be done via…

client.admin().indices().
               prepareAliases().
               addAlias("mynewindex", "myalias").
               removeAlias("myoldindex", "myalias").
               execute().
               actionGet();

And to obtain the index for a given alias, do:

ImmutableOpenMap<String, AliasMetaData> iom=
               client.admin().
                      cluster().
                      state(new ClusterStateRequest()).
                      actionGet().
                      getState().
                      getMetaData().
                      aliases().
                      get("myalias);

if (iom==null) // alias not found.

Iterator<ObjectObjectCursor<String, AliasMetaData>> 
                                      i=iom.iterator();

while(i.hasNext()) {
    ObjectObjectCursor<String, AliasMetaData> ooc=i.next();
    System.out.println("Index="+ooc.key+"/Alias="+
                          ooc.value.getAlias());
}

There are many ways to fill an index and the basic API for bulk filling is explained in the official documentation. I’ll cover more complex cases in another article.

When it comes to querying ElasticSearch indices, we face an entire new science :) The basic commands are explained in the official documentation, but the ways to obtain what you want from ElasticSearch are just legion.

Covering the possibilities of querying indices goes far beyond the scope of this article. After all, ElasticSearch provides an access point to the underlying (hard working) Lucene framework, a full text indexing and searching system that come now to its 15th year of existence.

Another aspect that I wanted to point out is that reading the source code of an open-source project, even as complex and big as ElasticSearch, is nothing magic. It does not only help to employ the product, but it also gives deep insights that the documentation often does not provide. Moreover, and that’s a very important note, reading and understanding the source codes brings you one step closer to contributing to the project!

Et zou…

European Ada Awards 2014 – Digital Girl of the Year

The year 2014 is the second year in which the EU Ada Awards are appointed to outstanding women and girls active in STEM and to organisations for promoting girls and women in STEM (Science, Technology, Engineering and Mathematics).

When I heard about the second edition, I directly thought about all the amazing girls that I met at the different Devoxx4Kids events that I attended during the last years (4x Belgium, 2x Paris, London, Geneva).

I would really love to see “our” girls participating in this award nomination. Getting rewarded and recognised with such a title is certainly a game changer in the life of a girl and being nominated can be one as well.

The last year’s winners have been invited to Vilnius, Lithuania, to Europe’s largest ICT event, ICT 2013, where Ms. Neelie Kroes, Vice President of the European Commission responsible for the Digital Agenda, announced the results.

With this blog entry, I would like to ask the Devoxx4Kids organizers and volunteers around the EU, if they could think of girls that showed amazing passion and skills during your events. We don’t need Supergirls – we are looking for passion, curiosity and projects! If you find someone, please don’t hesitate to kindly contact the parents for proposing a nomination.

If you are a parent of a girl that fits the above description – Devoxx4kids attendee or not, please consider going for a nomination too.

Of course, everything is voluntary and solely the decision of the parents and the kids. The nomination must be carried out by the parents. An unforgettable adventure awaits you and your girl! This year, the ceremony will happen in Italy btw.

If you need support, for example with language or submission, or if you have more questions, I’m glad to help!

More info:
http://adaawards.com/
http://digitalgirlaward.com/
https://www.facebook.com/TheAdaAwards

(Final Note: The Ada Awards are NOT related to the Devoxx4kids organisation).

First Devoxx4Kids event in London

The very first edition of Devoxx4Kids in the UK took place 1 week ago, on 14st of June 2014 in London.

Organising such an event just after a conference, the DEVOXX UK in this case, was quite challenging in terms of stamina. A good English breakfast “deluxe” provided the necessary energy to pass the day. I learned that I actually like beans for breakfast, how strange is that? But I’m diverting…

One should know that the Devoxx4Kids project is actually independent from the “big and famous” DEVOXX and therefore needs to find its own founding. Hence, since it is not easy to find a decent and not too expensive location in London, we settled at the Shoreditch Village Hall, which kindly provided us with space and equipment for this occasion.

Dan, the D4K UK project manager, opted for letting the kids choose at each round what workshop they would like to attempt. Usually, we schedule events in advance for optimal distribution and fun, so this time, we had quite some variations in the number of attendees per session. Moreover, the venue was one big hall with different workshop booths,  which made this event more a D4K hackathon for kids. This was actually not so bad seeing the amount of noise the 35 kids and the trainers produced :)

The ratio girls/boys was around 1 of 3.

Dan Hardiker (click), D4K UK project manager and DEVOXX UK co-organisator, doing the welcome talk…
The Oracle team interviewing Dan…
Mindcraft hacking with Arun (click) and his son…
Dave Snowdon (click), NAO developer, Devoxx UK speaker and NAO Conference co-organiser, teaching kids how to program a real robot…
Audrey Neveu (click), TEDx and DEVOXX UK speaker, teaching mindstorms…
Andy Piper (click), Twitter guy and Devoxx UK speaker, working the Arduino workshop…
An act of confidence for one of the kids :)
NAO doing his face recognition and tracking mission… always a fascinating thing for the kids…

Video clip with interviews recorded by the Oracle Technology Network team:

More pictures on the Facebook page and last but not least, Roy’s great video:

Devoxx4Kids Geneva – May 2014

The 17th of May 2014, we went to Geneva, Switzerland, to give our NAO workshop at the local D4K event. It was a long and a short stay, but we met amazing passionate people and about 60 nice and cool kids of different ages (the pictures shows only the 10-14 years old kids). A really cool experience!

Now, the pictures of the event are online and you can have a look by going here: https://www.flickr.com/photos/devoxx4kidsch/sets/72157644680905816/

How We Won The Battle On Net Neutrality In Europe

At the re:publica conference of 2014Thomas Lohninger  and Joe McNamee (EDRI) gave an overview on the fight for Net Neutrality at European level, named How We Won The Battle On Net Neutrality In Europe.

The EU Parliament has recently, and surprisingly, decided in favour of implementing a strong support for net neutrality in the EU legislation.

Many of us may think that the fight is over now, but on the contrary. This fall, the EU Council has yet to give its green light. You may know that the EU Council is actually a representation of the heads of state or heads of government of the EU member states  – thus it’s not the most democratically elected  and transparently working institution!

Brussels is the main “lobby city” in Europe and the high profile Telecommunication companies are already throwing all their lobby resources into the game. Despite of the decision of the EU Parliament, the situation actually does NOT look really good, especially since nobody seems to care.

At their talk at re:publica, Thomas and Joe provided a good overview on what we have to do next:

For several years the debate about net neutrality is going on in the EU. For a long time nothing happened besides consultations after consultations until September 2013 the European Commission suddenly proposed a regulation that claims to complete the digital single market and thereby abolish net neutrality almost completely. With only a few months left till the European elections and a very tight time schedule for political debate a coalition of several European NGOs started the campaign SaveTheInternet.EU. Its goal was to get the public informed about this legislation and get them to join in the fight for an open and free internet. At the time of this talk we will now how the fight ended so its time to look back and learn. This talk will either tell the story of how we won or why we lost net neutrality in Europe.

(Via Netzpolitik.org)

And to conclude with something cool, have a look at this (subtitles available) … #apartnet #apathy

Update:
Interesting read: A Neutral Guide to Net Neutrality by Stephanie Crets

Independent Socket server on NAO

For my latest project, the NAO for Scratch 2 extension, I wrote a network server that runs as NAO Choregraphe behavior on the robot.

As soon as the behavior is started, it listens permanently for an incoming connection until the behavior is stopped or the robot is shut down. When the client disconnects (gracefully or not), the server simply awaits a new connection. In this special case, I made my live simpler by allowing only 1 single concurrent connection. Having to deal with more parallel connections would invoke Thread management and that’s something I like to keep for later ;) In my use case, 1 connection is just fine.

So here is the graphical representation of the box:

SocketServerBox

The server behavior starts on the “OnStart” signal and sends an “OnStopped” when shut down. This means that the behavior is stopped and not that there is a disconnection. In any case, the sockets are closed properly inside the behavior.

Each time a client connects or disconnects, signals are sent via “OnConnect” or “OnDisconnect” respectively. This helps informing the user on what is going on.

Once connected,  the server awaits incoming command chains having the following format: “uid#command#param1#param2#…#paramN$“. “#” is the chunk separator, “uid” a unique command ID that is optionally used to inform the client that the command has finished to be executed (eg. think of an animation), “command” is the effective action command and “param1-N” an optional list of parameter values. “$” is the End Of Command control character. An example would be “123#say#Hello World!$“.

Using this simple pattern is a choice that allows us to test the server using a simple (raw-mode) telnet session. You may however send whatever you like depending on your application and use cases.

Once the command is received, it is transformed into a “string[10]” array and send over the “command” output to a handler box: “String[uid,command,param1,param2,…,paramN]“.

The choice to use an array to send the command data instead of separate output signals à la “command“, “parameter1“, “parameter2” etc., has been made to avoid race conditions on the signals. This avoids that, for example, the “command” signal is received with the “parameterX” signal still onits way. In Choregraphe, the order of reception is not guaranteed to be the same as the order of sending signals. The drawback of this solution is that the executor box needs to know the array structure, which is something that would not be required in a clean design (low coupling, high coherence). However, no choice in this case.

If a feedback to the client is desired, the executor box simply has to send the “string[10]” array back to the “OnCommandFinished” input to trigger the emission of a message of type “uid#end$” to the client.

The “uid” is always required since, even if we allow only 1 connected client at the same time, we permit to execute commands in parallel (like say and an animation). This means that the first command sent is not necessarily the first command to conclude.

And here is the code:

import socket

class MyClass(GeneratedClass):

    # ---------------------------------------------------------
    def __init__(self):
        GeneratedClass.__init__(self)
        self.listenSocket=None
        self.connectedSocket=None
        self.doingUnload=False
        pass

    # ---------------------------------------------------------
    def onLoad(self):
        pass

        
    # --------------------------------------------------------
    # Handles a single connection by receiving a message and decomposing it into fields,
    # which are then sent to an box output.
    #
    # Format incoming command string: "uid#command#param1#param2#...#paramN$",
    #
    # Format of the outgoing command array:
    #     String[uid,command,param1,param2,...,paramN]
    #
    def handleConnection(self):

        total_data=""

        try:
            while True:
                data = self.connectedSocket.recv(1)
                if not data:
                    self.logger.info("Socket "+str(self.connectedSocket)+" closed by peer.")
                    self.onDisconnect()
                    break
    
                if data=='$':
                    self.logger.info('Execute:'+total_data)
                    
                    # Convert the command string to an array of strings.
                    # We use an array instead of separate output
                    # signals to avoid race conditions on the 10 different signals
                    self.command( total_data.strip(' \t\n\r').split('#') )
                    
                    total_data=""
                    
                else:
                    if len(total_data)<255:  # data buffer limitation
                        total_data += data
                    
        except Exception as ex:
            self.onDisconnect()
            self.logger.info("Exception while listening on socket "+str(self.connectedSocket)+", close connection! ")
            self.logger.info(ex)
            
        try:
            self.connectedSocket.close();
        except:
            pass
        
        self.connectedSocket=None
        pass    
        
        
    # ---------------------------------------------------------
    # Starts the TCP/IP listener on the configured port and delegates the connection to a
    # handler. No multiple connections supported so far to avoid complicating things with
    # multi-threading and synchronised data management.
    #
    def onInput_onStart(self):
        
        while True:
        
            # Open port for listening
            try:
                self.logger.info("Listening on port "+str(self.getParameter("port"))+" allowing 1 concurrent connection.")
                
                self.listenSocket=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
                self.listenSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
                self.listenSocket.bind(("",self.getParameter("port")))
                self.listenSocket.listen(1)
                self.connectedSocket,address=self.listenSocket.accept()  # -> waiting for incoming connection
            
            except Exception as ex:
                # Very probably triggered by UNLOAD by stopping the behaviour
                if self.doingUnload==False:
                    self.onUnload()  # |_ don't call it twice
                break
        
            # Handle incoming connection
            self.onConnect()
            self.logger.info('Connected to '+str(address)+"/"+str(self.connectedSocket))

            # Send back a welcome message. The ~ prefix indicates a small talk message (i.e. can be ignored by client)
            self.connectedSocket.sendall("~Hello, NAO4Scratch speaking! Waiting for your commands!\n\r")

            self.handleConnection()
    
        # |_ while True
        pass

    # ---------------------------------------------------------
    # Stop the socket server box.  Closes all sockets and set the onStop output.
    #
    def onUnload(self):
        self.logger.info('Unloading: closing all sockets')
        
        self.doingUnload=True
        
        try:
            self.connectedSocket.shutdown(socket.SHUT_RDWR)
            self.connectedSocket.close()  
        except:
            pass
            
        try:
            self.listenSocket.shutdown(socket.SHUT_RDWR)
            self.listenSocket.close()  
        except:
            pass
                    
        self.connectedSocket=None
        self.listenSocket=None
        
        self.onStopped()
        pass

    # ---------------------------------------------------------
    # External signal forcing to stop the box.
    #
    def onInput_onStop(self):
        self.onUnload()
        #~ it is recommended to call onUnload of this box in a onStop method,
        # as the code written in onUnload is used to stop the box as well
        pass
        
        
    # ---------------------------------------------------------
    # External signal when the blocking command has been finished. We need to notify the
    # client by sending back the UID of the command.
    #
    # Format:  "UID#end$"
    #
    def onInput_onCommandFinished(self,cmd):
        try:
            self.connectedSocket.sendall(cmd[0]+"#end$")
            
        except Exception as ex:
            self.logger.warn("Unable to send back command complete notification")
            self.logger.warn(ex)
            
        pass

You find the box in its natural habitat in  the NAO4Scratch repo on GitHub. Note that you still find a lot if “self.logger.info” statements in the code. These are present in order to have a complete overview over the behavior as long as the project is in alpha/beta state. They will be replaced by debug output later.

DEVOXX Paris 2014

April, 16th anno 2014! Time again for the French edition of the DEVOXX conference, as usual in the city of lights: Paris.

Normally, this event is my “relax and enjoy” conference. No work, just discussions with cool people, seeing friends, making new acquaintances and attending some sessions. But this year was different!

Since the members of the Devoxx4Kids France team in Paris are also members of the DEVOXX France team, the organisers asked the Belgian D4K team to conduct a session on Wednesday.

Even having “only” 10 “younger” kids attending this single-session event, there was a huge “visibility” effect for D4K. At least 500 geeks crossed our group of mini-geeks while we were waiting for the D4K event to get started. Nice and fun :-) During the session, we worked with Scratch and the Leapmotion sensor, Lego Mindstorms EV3 and the NAO robot.

Kids having fun with the Lego Mindstorms EV3
Kids having fun with the Lego Mindstorms EV3
LeapMotion movement  detection and Scratch, a great combination!
LeapMotion movement detection sensor and Scratch, a great combination!

Another difference this year: the DEVOXX France conference had also a “Future<Devoxx>” track and part of this track was a hacking zone – The Laboratory – where makers and tinkerers presented their innovations and projects  – and everybody was invited to join hacking in the divers corners.

Laser organ...
Laser organ…
And floppy drive band :)
And floppy drive band :)

In this context, I was scheduled, together with Audrey Neveu and Xavier Bourguignon, to carry out a mini-hackathon on how to integrate the NAO Robot with Scratch 2. We wanted to allow even younger kids to program the robot.

We did not manage to have a full fledged prototype, but we got a running proof of concept, which allowed us to advance on this project during the weeks after the conference. “Success!” I would say!

NAO showing us how to connect him to Scratch :)
NAO showing us how to connect him to Scratch :)

Conclusion: It was a great conference for me. We started a very nice and useful FOSS project, we held a fun D4K session, I met a lot of interesting people and beside of this, I learned a lot, again, even not attending much sessions this year.

First Devoxx4Kids event in Paris, April 2014

On the 5th of April 2014, Pierre and I went to Paris in order to participate on the first ever Devoxx4Kids Paris event.

The Devoxx4Kids France coordination team, Audrey and Claude, were very busy during the the past year traveling France (and Switzerland) to make events all over the French speaking part of Europe. In their hometown Paris however, this was the first event they organised and we are very happy to be part of the party.

For us, this was the first D4K event with kids between 5 and 8 years old and kids between 8 to 10 years. Both groups had a half-day event with Makey Makey‘s (and 20kg of bananas), the Robot Turtles board game, WeDo‘s and Scratch. The group of 10 to 14 years followed the usual Scratch, Greenfoot, Arduino, Mindstorm and NAO workshop program.

During the happening, the students of E Artsup Nantes did a video recording and interviews with some kids. You can see the resulting video hereunder.

To conclude this little report, let’s have a look to some pictures:

This picture illustrates very well the spirit of D4K. Tweet by @morlhon.
Ancienne Secrétaire d’Etat de l’Economie Numérique Française, Madame Fleur Pellerin visiting the event.

 

NAO presenting himself!
Bananaaaaaaas, Heuheu!

A more complete photo gallery can be found here.

(Note: All pictures of the kids were taken with the signed consent of the parents)

(PS: “Bananaaaaaaas, Heuheu!” (c) The Minions)

Avoid Choregraphe dropping connection to NAO when running in a VM

The easiest way to provide a defined work environment when organising a Hackathon around the NAO robot is to prepare a Virtual Machine with all software preinstalled (Choregraphe, git, documentation, Python, etc.). Personally, I use Mint Linux  running in a Virtual Box (Ubuntu gives me this bad Google/Facebook feeling).

While installing Choregraphe is easy, you will quickly get blocked by the repeated disconnection of Choregraphe from NAO.

Normally, this indicates a firewall problem, but neither deactivating the firewall with “sudo ufw disable“, nor emptying the iptables defaulting it to “all open” with “sudo iptables -F“, nor explicitly defining “all open” rules with “sudo iptables -P INPUT ACCEPT / sudo iptables -P OUTPUT ACCEPT / sudo iptables -P FORWARD ACCEPT” helps preventing the disconnections.

Still, the reason is a port blocking problem. While Choregraphe is able to connect to a target outside the VM’s host operating systems, inbounding connections reaching the host systems are not reaching the VM’s guest system.

This leads us to the easy solution: defining a port forward rule from the host OS to the guest VM.

In Virtual Box, this is done via the VirtualBox main menu “Devices”, “Network”, “Network Setting”, Adapter 1, [Port Forwarding]. Here, we add the Choregraphe’s main ports (54010-540nn, depending how many instances you have running) and the most important: the NAOqi port 9559.

If you are not sure about the ports, simply display the debug log files in Choregraphe to make the concerned ports visible (see screenshot above).

Once the port forwarding confirmed, the connection with the NAO root should be stable, without even restarting the VM.

Jake Appelbaum reads his Homeland afterword

The net activists Aaron Swartz (RIP) and Jacob Appelbaum contributed to Cory Doctorow‘s book Homeland by writing an afterword on freedom and all-present surveillance. Now, Jacob’s text has been recorded as part of the audio book release and it has been published by Cory Doctorow on his blog.

Just take 6 minutes of your time and listen to what he had to say. I provide the Alec Empire‘s vocoder mix, which simply sounds mega cool :-) The original is available on Cory’s blog post.

More audio extracts (read by Noah Swartz and Wil Wheaton) are available here: http://craphound.com/homeland/

License: Copyright CorDoc-Co, Ltd (UK), 2013.;\\ License: Creative Commons Attribution-NonCommercial-NoDerivs 3.0

.

Afterword by Jacob Appelbaum, Wikileaks

“Utopia is impossible; everyone who isn’t a utopian is a shmuck.”

Cory asked someone to write to the children of the newest generation and to say something to inspire them. To write something that would encourage them to take up the cause of bettering the world. That’s you or someone you love – when you’re finished, please pass this book on to the person who needs it most.

Everything good in the world comes from the efforts of people who came before us. Every minute that we are able to enjoy in a society that is not ruled by senseless violence is a minute given to us by the hard work of people who dedicated their lives for something better. Every person we meet is carrying his own burdens. Each person is the center of her own universe. There is so much left to be done, so many injustices to right, so much suffering to relieve, so many beautiful moments to be lived, an endless amount of knowledge to uncover. Many secrets of the universe wait to be uncovered.

The deck from which our hands are dealt need not be stacked against us; it is possible to create societal structures that are just and capable of reasoned compassion for everyone. It is possible to change the very nature of our lives. It is possible to redesign the entire deck, to change the very face and count of the cards, to rewrite the rules and to create different outcomes.

We live in the golden era of surveillance; every phone is designed to be tapped, the Internet passes through snooping equipment of agencies that are so vast and unaccountable that we hardly know their bounds. Corporations are forced (though some are willing enough!) to hand over our data and data of those whom we love. Our lives are ruled by networks and yet those networks are not ruled by our consent. These networks keep us hooked up but it is not without costs that they keep us hooked together. The businesses, the governments, and the individuals that power those networks are incentivized to spy, to betray and to do it silently. The architecture of the very systems produces these outcomes.

This is tyranny.

The architecture of our systems and of our networks is not the product of nature but rather the product of imperfect humans, some with the best of intentions. There is no one naturally fit to survive in these unnatural systems, there are some who are lucky, others who have adapted.

This letter to you, from your perhaps recent past, was written with Free Software written as a labor of love by someone who wished to help the children of Uganda while flying over an expansive ocean at difficult to understand heights, it was composed while running under a kernel written by scores of people across every national line, across every racial, sexual and gender line by a socially and politically agnostic engineer, it was sent through multiple anonymity networks built by countless volunteers acting in solidarity through mutual aid and it was received by an author who published it for a purpose.

What is the common purpose of all of these people? It is for the whole of our efforts to be more than the sum of our parts — this creates a surplus for you — to give breathing room to others, so that they may take the torch of knowledge, of reason, of justice, of truth telling, of sunlight — to the next step, wherever it may lead us.

There was a time when there were no drone killings, societies have existed without armed policemen, where peace is not only possible but actually a steady state, where mass surveillance was technically and socially infeasible, where fair and evenhanded trials by impartial juries were available for everyone, where fear of identification and arrest was not the norm but the exception. That time was less than a generation ago and much more has been lost in the transition from one generation to the next.

It’s up to you to bring those things back to our planet. You can do this with little more than cooperation, the Internet, cryptography, and willingness. You might do this alone or you might do it in a group; you might contribute as a solitary person or as one of many. Writing Free Software empowers every person, without exception, to control the machines that fill our lives. Building free and open hardware empowers every person, without exception, to construct new machines to free us from being slaves to machines that control us. Using free and open systems allows us to construct a new basis by which we may once again understand as a whole, the systems by which we govern ourselves.

We are on the edge of regaining our autonomy, of ending total state surveillance, of uncovering and holding accountable those who commit crimes in our names without our informed consent, of resuming free travel without arbitrary or unfair restriction. We’re on the verge of ensuring that every person, not one human excluded, has the right to read and the right to speak. Without exception.

It’s easy to feel hopeless in the face of the difficult issues that we face everyday — how might one person effectively resist anything so much larger than herself? Once we stop acting alone, we have a chance for positive change. To protest is to stop and say that you object, to resist is to stop others from going along without thinking and to build alternatives is to give everyone new choices. Omission and commission are the yin and yang of personal agency.

What if you could travel back through time and help Daniel Ellsberg leak the Pentagon papers? Would you take the actions required, would you risk your life to end the war? For many it is easy to answer positively and then think nothing of the actual struggles, the real risk or the uncertainty provided without historical hindsight. For others, it’s easy to say no and to think of nothing beyond oneself.

But what if you didn’t need to travel back through time?

There are new Pentagon Papers just waiting to be leaked; there are new wars to end, new injustices to make right, fresh uncertainty that seems daunting where success seems impossible, new alternatives need to be constructed, old values and concepts of justice need to be preserved in the face of powerful people who pervert the rule of law for their own benefit.

Be the trouble you want to see in the world, above nationalism, above so-called patriotism, above and beyond fear and make it count for the betterment of the planet. Legal and illegal are not the same as right and wrong — do what is right and never give up the fight.

This is one idea out of many that can help you and your friends free our planet from the tyranny that surrounds us all. It’s up to you now – go create something beautiful and help others to do the same. Happy hacking,

Anonymous
000000/002012/00/00/00:00:00:00