Strange exceptions with Objectify 4, or: what happens if you query unindexed fields

When working with the Google Appengine Datastore, Objectify is the framework to use. Currently it is available in version 4 and the API is simple and beautiful.

On important rule when working with the Datastore – regardless of the framework you use – is that all fields that are used in a query filter have to be indexed. But what happens if you forget to add the @Index annotation?

Here is a simple example:


@Entity
public class Person {
    @Id private Long id;
    private String name;
    private final Collection interests = new ArrayList<>();

...
}

Let’s run a quick test:


    private final LocalServiceTestHelper helper = new LocalServiceTestHelper(new LocalDatastoreServiceTestConfig());

    @Before
    public void setUp() {
        helper.setUp();
    }

    @After
    public void tearDown() {
        helper.tearDown();
    }

    @Test
    public void testFindByName() {
        ObjectifyService.register(Person.class);

        Person p = new Person();
        p.setName("Max");
        ofy().save().entities(p).now();

        List persons = ofy().load().type(Person.class).filter("name", "Max").list();
        assertEquals(p.getId(), persons.get(0).getId());
    }

What do you think will happen? Yes, an exception is thrown:


java.lang.reflect.UndeclaredThrowableException
	at $Proxy12.get(Unknown Source)
	at org.cloudme.Person.testFindByName(Person.java:67)
...
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:32)
	... 27 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(ArrayList.java:604)
	at java.util.ArrayList.get(ArrayList.java:382)
	... 32 more

How do you fix it? Just add @Index to the name field.

Now, lets have another test and query on a collection field:


@Entity
public class Person {
    @Id private Long id;
    @Index private String name;
    @Index private final Collection interests = new ArrayList<>();

...
}

    @Test
    public void testFindByInterest() {
        ObjectifyService.register(Person.class);

        Person p = new Person();
        p.getInterests().addAll(Arrays.asList("Coding", "Jogging", "Reading"));
        ofy().save().entities(p).now();

        List persons = ofy().load().type(Person.class).filter("interests", "Reading").list();
        assertEquals(p.getId(), persons.get(0).getId());
    }

What happens now? The same thing! How is it possible? The reason is, that final fields are not persisted and therefore you can’t query them as well.

Summary: if you start making “stupid” mistakes, forgetting some annotations, making fields final etc., then you might come across some exceptions that don’t really tell you what your problem is. As a rule of thumb: whenever you have problems querying, check if the fields are indexed.

Notepad Bookmarklet

My latest application – Notepad - is not yet ready for prime time. If you dare, you can use it as a test user, but you will risk to lose data and have some incomplete features. However, I already created a bookmarklet for the application in order to run it smoothly inside your browser:

Drag this to your browser’s bookmark bar: Notepad

An jQuery AJAX framework for mobile request based webapps

In the previous posts I have described roughly the idea which I implemented in order to learn some new techniques and a short architecture overview of the web application. Now I want to write a little bit about the framework Sugar JS which I developed for this application. It shows how easy it is with jQuery to write some custom AJAX behavior for your own application.

Before I start, let’s have a short walkthrough to the application. Initially it starts with an empty screen:

You see the title bar (“LISTS”), a grey text and a footer with the user information and a link to log out. The grey text reads “Create a new list”, and is in fact a text field in which you write the application name. Notice: no submit button, no borders, just plain text. This minimalistic design is by intention – maybe not the prettiest, but I like it (maybe I’ll change it later). But I wanted to let the user tap as less as possible, therefore there is no submit button. Once the user entered the text, the list gets created.

When tapping the list, its contents will be shown and can be edited. Same here: no submit buttons to reduce number of taps for the user. Here is a screenshot of an example list in edit mode:

You can see items added to the list (Apples, Milk) and raw items (Cheese). Items can have attributes (“2 bottles”). The idea behind this is: reuse items to avoid too much typing, but be flexible enough to change attributes of items.

Items can be deleted by swiping horizontally, which displays a delete button to the item, and to be more safe, a confirmation dialog is displayed when tapping on the delete button:

All this is handled by the Sugar JS framework. It provides:

  • Display of a confirmation dialog on certain links; the selector can be specified for the links which should display the confirmation dialog. One example would be to use all links with the class “confirm”. When displaying the confirmation dialog, the text of the link’s title is shown.
  • Handle form input and automatic submit when the user leaves the form
  • Handle synchronous and asynchronous requests. The current implementation interprets all URLs, which start with a “!” as asynchronous request, regardless if it is in a link or a form action. This allows entering an attribute without reloading the page, as an example. It is also used when tapping on the items in the list (“tick”). This works quite well currently, but might get changed to a selector or class based solution later (e.g. all links/forms with class “async” will be handled asynchronously.
  • Automatically extend text fields even while typing text.

All this is done inside of a static (i.e. non JSP) HTML page. To use Sugar JS, you need to include the sugar.js script and attach it to an element of your document. It can be the body element or any other element:

$(document).ready(function() {
  $('body').sugar({ 
    url : '/action/index',
    submit : 'input.submit',
    confirm : 'a.delete',
    expand : 'input.attribute',
    onLoad : function() {
        $('a.checkin').click(checkin);
        $('a.unticked').click(tickItem);
        $('a.edit').click(toggleInNote);
...
});


The following initialization takes place:

  • $('body').sugar(...); initializes the Sugar JS framework and binds it to the body tag. The whole contents of the body tag will be replaced.
  • url : ... specifies the initial URL that is loaded into the tag, which the framework is bound to.
  • submit : ... specifies the selector of the elements that are enabled for automatic form submit. In this example all input fiels with class “submit”.
  • confirm : ... specifies the selector of elements that show a confirmation dialog when clicked (e.g. before deleting).
  • expand : ... specifies the selector of elements that are expanded while typing.
  • onLoad : ... is the place where custom Javascript will be added, which is executed after each synchronous load. This is required, to bind event handlers to DOM elements, which are have been loaded.

The custom functions you see (e.g. tickItem) mostly do some changes to CSS attributes to reflect changes on the screen based on user actions. But they can also handle more complex logic, such as the checkin function, which sends the geolocation to the server. My current implementation is quite a hack (and cries for improvement of the Sugar JS framework):


var checkinLink = undefined;

function checkin(event) {
  event.preventDefault();
  if (checkinLink == undefined) {
    event.stopImmediatePropagation();
    checkinLink = this;
    navigator.geolocation.getCurrentPosition(function(position) {
      var latitude = position.coords.latitude; 
      var longitude = position.coords.longitude;
      doCheckin(latitude, longitude);
    }, function(error) {
      doCheckin(0, 0);
    });
  }
  else {
    checkinLink = undefined;
  }
}

function doCheckin(latitude, longitude) {
  var position = latitude + '/' + longitude;
  var url = $(checkinLink).attr('href') + position;
  $(checkinLink).attr('href', url);
  $(checkinLink).click();
}


The problem with geolocation is: a) it is implemented asynchronously (so you cannot just say simply something like var position = getGeoLocation(), but instead you have to pass a function which gets called by the browser, and b) somehow the location has to be passed to the server, which I have handled by changing the URL of the link that has been clicked (yep, dirty hack) and then clicking it again to trigger the Sugar JS framework. I’m sure it can be done much better.

[Update] Now you might ask, what all this has to do with “request based” webapps. Now this is the deal with Sugar JS: Stripes Framework is request based, which means it handles only full page requests. If the user clicks a link, the whole page gets rendered. If the user submits a form, the same happens. Just by attaching Sugar JS to the page, it handles all synchronous and asynchronous requests.

The initial page is index.html. It is a normal HTML page with some Javascript (see above) and the body element looks like this:

<body>
</body>


It is empty. It will be populated by the Sugar JS framework on each (synchronous) request. The initially loaded page is configured by the url parameter (see above). Since Stripes provides some templating, here I have written an iphone.jsp template, which provides header, content and footer. It gets referenced by all other pages, such as index.jsp, note.jsp and edit.jsp. These pages are normal JSP pages without any Javascript; example: index.jsp:

<%@ page pageEncoding="UTF-8" contentType="text/html; charset=UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="s" uri="http://stripes.sourceforge.net/stripes.tld"%>
<s:layout-render name="/layout/iphone.jsp">
<s:layout-component name="title">Lists</s:layout-component>
<s:layout-component name="content">
<ul class="edgetoedge">
<c:forEach items="${actionBean.notes}" var="note">
<li><a href="/action/note/checkin/${note.id}/" class="checkin">${note.name}</a></li>
</c:forEach>
</ul>
<form action="/action/index/create" class="edgetoedge">
<input type="text" name="note.name" placeholder="Create a new list" class="submit"/>
</form>
</s:layout-component>
</s:layout-render>

Mobile Webapp Architecture – Overview

In my previous post I described the idea of a small mobile webapp. In this article I will briefly describe the architecture of the application.

The main goals of the architecture are:

  • Maintainability – the application should be clearly structured and understandable
  • Portability – although it is currently deployed to Google App Engine, it should be easy to port it to a different platform
  • Performance – the application should be fast and lightweight; considering the characteristics of Google App Engine, it should have a short cold start time

Here is an diagram showing the architecture of the webapp (click to enlarge):
Loclist Architecture Diagram

The application consists of 4 services:

  1. The UserService manages user data and multitenancy. Both is provided by the GAE environment, so the user service itself is quite small.
  2. The ItemService manages the items. An item can be reused in multiple lists (called “notes” in my application).
  3. The NoteService handles items in a note (a list), such as adding and removing items to a note and ticking items.
  4. The LocationService manages locations and sorting of items in a note depending on the actual location.

Each service encapsulates functions, entities and data access. Services can interact with each other, but direct access to data access objects of other services is prohibited. More on this in a later post.

The main client platform is the iPhone (although it should run on Android devices as well), so the application is designed to run as a fullscreen webapp started from the iPhone’s home screen. Sometimes, links are opened in Safari when using fullscreen mode. To avoid this, all links are handled by the AJAX framework Sugar JS, which I developed for this application. It is based on jQuery and allows development of a normal request based web application (such as Stripes Framework, which is request based), and have an AJAX frontend.

On a side note: is this architecture over-designed for this kind of application? Certainly it is. But the intention of this application was not be be ready as quickly as possible, but to practice some techniques which I haven’t looked at recently in detail.

New mobile webapp: Localized Lists

Recently I finished developing a new mobile webapp. The primary goal was to update my experience with some technologies and the secondary goal was to implement an idea.

First I’ll tell a bit about the application itself. The project name is “Loclist” which is a combination of “Localized” and “List”. While applications such as grocery lists are ten a penny, my application is a little bit different. Sure, it provides the basic features a grocery list needs: the user defines a set of items and adds the items to the list. Each item can have an attribute (such as “2 red Apples” – “2 red” is the attribute and “Apples” is the item). But my application is a little bit different: it learns based on the location, to sort the items in the list. This way, the list learns “path” you take shopping in a super market and presents the grocery list in the best order. You can try it for yourself, the application is free for everyone, just log in with your Google account here: http://lists.cloudme.org.

But this article is not about marketing the application; instead I’d like to write about the technology I used for creating the application. As it is an experiment, it should be a low-budget project. Therefore a starting point was Google App Engine (GAE), which provides free webspace (until some quota limits kick in). Unfortunately, GAE has some restrictions. In order to make the application fast, it should be quite lightweight. That means, care must be taken to choose the right libraries and frameworks to work with. Another constraint was, that it should be a mobile application, which should run almost like a native application. On the iPhone, it should be possible to run in fullscreen mode, when launched from the home screen. So I chose the following technology stack:

All this together makes a very convenient platform to develop mobile webapps. But there are some best practices, which are not baked into this stack, and so I created a new, lightweight framework which glues together all these technologies. It is called “Sugar” and consists of two parts:

  1. Sugar GAE – the Java based framework which connects Objectify, Stripes and Guice to an easy to use platform and adds additional features, such as abstract classes for standard DAOs, Service classes, test cases and some Stripes Framework features.
  2. Sugar Web – the jQuery and CSS based framework that provides AJAX features to Stripes based web applications.

Some might ask, why I didn’t use an existing framework to develop iPhone webapps, such as jQTouch or jQuery Mobile. But that would have been too easy. First of all, I wanted to learn more details about writing my own jQuery plugin and designing my own iPhone webapp. But apart from that, my application does not exactly behave like a normal AJAX webapp. Instead, it requires to communicate the geolocation to the server in order to provide location based sorting of lists. Another feature wanted to learn. Therefore I decided to not use an existing framework. I was surprised to find out that writing such a framework was quite easy and required less then 80 lines of code.

With small improvements to better Google App Engine performance

While the Google App Engine (GAE) is a nice environment for developing and hosting Java web applications it has some constraints that should be considered during development. On the one hand, there are restrictions in the classes that can be used. GAE allows only a subset of the JavaSE classes to be used. This, however, is in my experience normally not a big issue. Even with this restriction it is possible to write good web applications.

Performance is another issue – and becomes more and more important. While GAE isn’t the highest performant environment anyway, it has some annoying characteristics that should be considered when developing web applications: the initial load request, also known as application cold start time. This is due to the fact that GAE instantiates a new VM for a web application if the application hasn’t been used for a while (the duration is not specified, but may be down to a couple of seconds). This instantiation takes some noticable time. Depending on the framework that is used by the application, it can take several seconds.

Per has a thorough analysis and provides some workarounds in the blog of his new project Small Improvements.

Simple confirm dialog with jQuery

The simple-confirm jQuery plugin adds a confirm dialog to links (or other elements). A dialog will be prompted to the user to confirm the link action. This might be useful for deleting items. To use the simple-confirm pulgin, you might have a link like this:


<a href="delete.php" class="confirm" title="Do you want to delete?">Delete</a>

To add the confirm dialog you need the simple-confirm plugin


<script type="text/javascript" src="js/simple-confirm-0.1.js"></script>

and use it:


<script type="text/javascript">
$(document).ready(function() {
    $("a.confirm").confirm();
});
</script>

Download simple-confirm-0.1

Unit testing Stripes ActionBeans

There are two ways for unit testing Stripes applications: directly instantiate the ActionBean or use Mock objects. But neither provided the “feel” I wanted to have for my unit tests.

In my tests I wanted to use a URL only to call the right ActionBean. So, my own solution is a mixture between mocking and direct instantiation. in my tests I use the following method to create an ActionBean:


    protected <T extends ActionBean> T createActionBean(String url, Class<T> clazz, Map<String, Object> params)
            throws Exception {

        T instance = clazz.newInstance();
        injectMembers(instance);

        UrlBindingFactory factory = UrlBindingFactory.getInstance();
        factory.addBinding(clazz, UrlBindingFactory.parseUrlBinding(clazz));
        UrlBinding binding = factory.getBinding(url);

        Method method = null;

        for (UrlBindingParameter parameter : binding.getParameters()) {
            String name = parameter.getName();
            if (name.equals("$event")) {
                if (parameter.getValue() != null) {
                    method = clazz.getDeclaredMethod(parameter.getValue());
                }
            }
            else {
                Field field = clazz.getDeclaredField(name);
                field.setAccessible(true);
                Validate validate = field.getAnnotation(Validate.class);
                Object value = null;
                if (validate != null) {
                    Class<? extends TypeConverter> converter = validate.converter();
                    if (converter != null) {
                        value = converter.newInstance().convert(parameter.getValue(),
                                field.getType(),
                                new ArrayList<ValidationError>());
                    }
                }
                if (value == null && parameter.getValue() != null) {
                    if (Number.class.isAssignableFrom(field.getType())) {
                        value = DecimalFormat.getNumberInstance().parse(parameter.getValue());
                    }
                }
                field.set(instance, value);
            }
        }

        if (params != null) {
            for (Map.Entry<String, Object> entry : params.entrySet()) {
                Field field = clazz.getDeclaredField(entry.getKey());
                field.setAccessible(true);
                field.set(instance, entry.getValue());
            }
        }

        if (method == null) {
            for (Method m : clazz.getDeclaredMethods()) {
                if (m.isAnnotationPresent(DefaultHandler.class)) {
                    method = m;
                }
            }
        }
        if (method != null) {
            method.invoke(instance);
        }
        return instance;
    }

What is still missing: I’d like to use the Stripes provided mock object classes more to support all types of parameters, validations etc.

Optimizing for GAE: Objectify and Guice

When I started designing my webgallery application, I chose standard frameworks in order to keep the application (potentially) portable with minimal refactoring effort. I was not totally focused on Google App Engine, instead I was ignoring typical characteristics of its architecture. That is a totally valid approach and has on its positive side the advantage, that you can almost completely write normal Java applications that run on GAE (except some standard classes cannot be used). However, a big disadvantage of this approach is, that the application cold start time was uncomfortably long, as a lot of libraries have to be loaded before the application starts.

Therefore, I removed all references to the Spring framework and the Datanucleus JDO implementation. The migration started on the persistence layer. Luckily I added a DAO layer to my design, so all I needed was changing annotations in the model classes and writing new DAO implementations (interfaces remained the same). When using Spring I made use of the “JdoSupport” classes, which provide convenience methods for using transactions etc. As I don’t use the full scope in my application, it was easy to implement the part that I actually needed in my own framework.

Then I removed all references to the Spring framework. My initial thought was that I just get rid of DI at all until I find a better solution, but that appeared to be not practical and testing started to get annoying. Therefore I decided to use Guice. Migration was pretty simple. I chose the approach to use member injection, so only @Inject annotations to the member classes are required. As Guice doesn’t use XML configuration, a Module is required, that contains the configuration. Sounds a bit strange at first glance, but it turned out to be very comfortable and actually very type safe (as it is configuration in a Java class.
Next part was integration in Stripes: I was looking for some available implementations but quickly found out that integrating it into my application was just a matter of a very small class – all I needed was dependency injection in my ActionBeans. So I decided to write that integration by my own:


@Intercepts( { LifecycleStage.ActionBeanResolution } )
public class GuiceInterceptor implements Interceptor {
    private Injector injector;

    public GuiceInterceptor() {
        injector = Guice.createInjector(new WebgalleryModule());
    }

    @Override
    public Resolution intercept(ExecutionContext context) throws Exception {
        Resolution resolution = context.proceed();
        injector.injectMembers(context.getActionBean());
        return resolution;
    }
}

The point is to write a Stripes interceptor that intercepts the lifecycle stage of ActionBean resolution – therefore the annotation of the class. The interceptor first creates the Guice injector in the constructor – with referencing to my module (that’s the benefit of using a custom implementation – I don’t have to care about too much configuration). The main part is the intercept() method, which lets the interceptor inject the members in the ActionBean.
The GuiceInterceptor just has to be placed into the extension package of my application in order to get recognized by Stripes (since version 1.5). That was already configured in the web.xml deployment descriptor:


    <init-param>
      <param-name>Extension.Packages</param-name>
      <param-value>org.cloudme.webgallery.stripes.extensions</param-value>
    </init-param>

The application runs overall faster now and uses a lot less external libraries than before. I can recommend everyone who implements for Google App Engine to try to reduce the application size as much as possible by removing unnecessary complex dependencies. With the right application design, switching between technologies can be very easy. My initial concern that, when using very GAE specific technologies (such as Objectify), I get stuck in that platform and can never escape, is irrelevant, as migration to a different technology can be very simple. Just ensure you have sufficient test cases.

Reinventing the wheel since 1973