Tuesday, July 20, 2010

Guice

NOTE: There is an intermediate stage for the project between the last posting and this one. In it, I add a servlet that will populate the datastore with sample data, optionally deleting the datastore first. It is protected by web.xml directives that no longer work after moving the servlet to Guice, but it's available as tag "add-objectify" on github.

Using Guice to inject an appropriate Dao is kind of a no-brainer goal. Use it to inject one that uses the App Engine datastore for production, and one that's got a dummy at the back end for testing.

Prepare to be Guiced


We need to make a Data Access Object interface that the methods needing it can use. The way that makes most sense to me is to rename our current DAO class to DaoGaeDatastore (correcting the capitalization to match the Java naming conventions), and extract from that a new Dao interface.

Eclipse makes this really easy to accomplish. The Refactor->Rename... menu item handles renaming the class, as well as updating all references to it in the code to the new name. The Refactor->Extract Interface... applied to DaoGaeDatastore pulls out the interface Dao, also with no real trouble. While I was at it, I removed the extra Objectify object I had added to the DAO, and used the ofy() method implemented by its parent DAOBase.

To the interface and implementation class, I added a method named deleteAllUsers(), which does exactly what you'd expect based on the name.

com/lisedex/volinfoman/server/DaoGaeDatastore.java
@Override
        public void deleteAllUsers() {
                ofy().delete(ofy().query(User.class).fetchKeys());
        }

One of the Objectify.delete() methods accepts an iterable collection of objects that can be a mix of Keys and POJOs (Plain Old Java Object). Executing a query() on User.class will return all Users, and the Query.fetchKeys() method will return an iterable collection of their Keys. We can just pass this collection to Objectify.delete(), and all of our Users in the datastore disappear.

We can now convert our BuildDB class, which was built between the last blog entry and this one, to use Dao.deleteAllUsers(), and remove its dependency on an Objectify object.

Let the Guicing commence


We need the guice-servlet jar.

I used the Guice integration documentation to see how to get Guice set up for the App Engine environment. All servlets where we want to use Guice need to be moved from their web.xml declarations into Guice module configurations. Also, all servlets must be have the Singleton scope, so we can either use the @Singleton notation in the class definitions themselves, or follow the design philosophy we used for Gin and declare them all in a Guice module. I chose the latter for consistency.

First we build an AbstractModule called VolinfomanGuiceModule. Its sole function is to override the configure() method to declare our Guice object bindings.

com/lisedex/volinfoman/server/guice/VolinfomanGuiceModule.java
public class VolinfomanGuiceModule extends AbstractModule {
        @Override
        protected void configure() {
                // Servlets
                bind(UserServiceImpl.class).in(Singleton.class);
                bind(BuildDB.class).in(Singleton.class);
        }
}

Since all servlets need to have Singleton scope, and we're not injecting them with replacement objects, this is all the binding we need.

If you've looked at the code before now, you'll notice we're using a library called gwt-log with a remote logging servlet that's been declared in web.xml. We're not moving this to the Guice configuration because it doesn't need Guice injection right now, and it also produces a warning when it's not declared in web.xml. This warning problem is corrected in r504 of gwt-log, but I'm only using the release version.

Next, we need a ServletModule that sets up what URLs each servlet handles, so we build VolinfomanServletModule and override configureServlets().

com/lisedex/volinfoman/server/guice/VolinfomanServletModule.java
public class VolinfomanServletModule extends ServletModule {
        @Override protected void configureServlets() {
                serve("/volinfoman/user").with(UserServiceImpl.class);
                serve("/volinfoman/admin/builddb").with(BuildDB.class);
        }
}

Each serve() call tells Guice which URL to map to which servlet class. Each servlet can handle multiple URLs through multiple calls to serve().

Finally, we need a GuiceServletContextListener that provides a Guice Injector that knows about the modules so it can configure the injection properly.

com/lisedex/volinfoman/server/guice/VolinfomanGuiceServletContextListener.java
public class VolinfomanGuiceServletContextListener extends
                GuiceServletContextListener {
        @Override
        protected Injector getInjector() {
                return Guice.createInjector(
                                new VolinfomanServletModule(),
                                new VolinfomanGuiceModule());
        }
}

In web.xml we need to remove references to the servlets we're configuring through Guice, and add the configuration for Guice in their place. After taking out the BuildDB and UserServiceImpl declarations, but leaving the RemoteLoggerServiceImpl for gwt-log, we put in the Guice code.

war/WEB-INF/web.xml
<!-- Configure Guice servlet, other servlets configured in
                 VolinfomanServletModule -->
        <filter>
                <filter-name>guiceFilter</filter-name>
                <filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
        </filter>

        <filter-mapping>
                <filter-name>guiceFilter</filter-name>
                <url-pattern>/*</url-pattern>
        </filter-mapping>

        <listener>
                <listener-class>com.lisedex.volinfoman.server.guice.VolinfomanGuiceServletContextListener</listener-class>
        </listener>

We set up a GuiceFilter with a URL pattern that describes which URL will be processed by Guice. We also plug our GuiceServletContextListener as a listener so Guice gets configured when the application is deployed.

Moving the BuildDB servlet to Guice configuration stops the authentication we had set up through a <security-constraint> in the servlet container. We'll have to add security in the code itself later, but for now the BuildDB servlet is open to the world.

Guiced!


All that set up for what will appear to be a pretty small payoff, but we're ready to use Guice anywhere we want in our servlets, as well as Gin in our GWT client.

We want to inject an instance of DaoGaeDatastore in place of fields declared as Dao in our servlets. First we need to configure this in VolinfomanGuiceModule.

com/lisedex/volinfoman/server/guice/VolinfomanGuiceModule.java
// Data providers
            bind(Dao.class).to(DaoGaeDatastore.class).in(Singleton.class);

After adding this to the configure() method, a DaoGaeDatastore singleton will be provided to any servlet that has a class field of type Dao that we also annotate with @Inject. So we need to inject it in UserServiceImpl.

com/lisedex/volinfoman/server/UserServiceImpl.java
@Inject
    private Dao dao;

This replaces the previous declaration for dao, and tells Guice to do the injection for that field. The rest of the class stays the same, since we were already using the Dao interface to define our interaction with the datastore.

In BuildDB we remove the local variable dao in doGet(), and use the same code as above to declare it as a class variable that needs injection.

That's it. The project is now Guicified.

Using Objectify to access the datastore

There are several options to choose from when deciding how you want to access the App Engine datastore. Two options are Java Data Objects (JDO) and Java Persistence API (JPA); both are from Sun and are in the Java SDK. From searching around the web, the consensus seems to be that both are too heavyweight for most App Engine projects. They are, however, both standards, and JDO is datastore agnostic, which would make moving your application off of the App Engine platform somewhat easier. However, you still are stuck with the limitations of the GAE datastore, so it's not quite so easy bringing JDO code into your GAE project.

Open source alternatives for Google App Engine


Two GAE specific open source projects are Objectify and Twig. Out of the two, Objectify seems simpler and more aligned with what's actually happening in the datastore. As I learn GAE, I feel it's somewhat imperative to not have the details of the datastore hidden from me. It's also modeled after the GAE Python library, which seems pretty clean. And, last but not least, your data classes can be passed through GWT RPC without modification. Perhaps in the future I'll come to a different conclusion if I get tired of having to write too much in the way of nuts and bolts that Twig handles automatically. Twig also supports parallel queries and merging OR queries, both of which could be pretty nice.

Hearing about the different design philosophies straight from the horses' mouths gave me some good information I needed to make this decision, too.

The Objectify concepts documentation is a good starting point for understanding both Objectify, and the GAE datastore.

Adding Objectify to VolInfoMan


Adding Objectify to the project leads us to the first time that we need a servlet running on App Engine. The servlet will simply provide a front end to Objectify and the datastore by responding to GWT RPC calls.

The code for this state of the project is available as tag "add-object-step1" on github.

He's not any kind of program, Sark. He's a User.


The first step is to define the object we want to pass back and forth between our GWT code and the GAE servlet. To continue the login page we've been working on, we'll need a User so we can authenticate the session. Since the User will be shared between GWT and GAE, we'll put it in the shared package.

(I'm leaving out the package and import statements to conserve space)

com/lisedex/volinfoman/shared/User.java
public class User implements Serializable {
        @Id
        private Long id;
        @Indexed
        private String username;
        @Indexed
        private long status;
        @Unindexed
        private String firstName;
        @Unindexed
        private String lastName;
        @Unindexed
        private String email;
        @Unindexed
        private String password;

Objects that will go through RPC need be Serializable. Later on, I'll probably add the @Cached annotation to the class so that it will automatically be cached by GAE's memcache.

You can only run queries on the GAE datastore against fields that are indexed. More specifically, you need an index for every query you intend to run, so by default all class fields are indexed to make it easier to query on any specific field. However, according to the Objectify best practices documentation, each property that is indexed requires a separate write to the index, which won't add to latency (writes are done in parallel), but will add to the CPU time used by the application. So instead, I explicitly declare which fields are to be indexed, and which are not. The same could be achieved by declaring the whole class @Indexed and only specifying the @Unindexed, or vice versa, but I like to explicitly declare each for easier reading. The field to be used to create the Key for the object is annotated with @Id.

I put in a couple of getter/setters, and a toString() method for easy logging. It will need to be fleshed out more later.

GAE, we need to have a serious talk. - Love, GWT


Next, we need to define the interface our client uses to access the servlet. I put GWT code for handling data into com.lisedex.volinfoman.client.data, so I define a UserService interface in that package.

com/lisedex/volinfoman/client/data/UserService.java
@RemoteServiceRelativePath("user")
public interface UserService extends RemoteService {
        User getUser(Long id);
        User getUser(String username);

        void putUser(User user);
}

The important part is that we declare what URL we'll be using to access the RPC servlet using the @RemoteServiceRelativePath annotation. The interface itself extends RemoteService, which all GWT RPC client interfaces should extend, and then we add the RPC functions we want to support. I want to support grabbing a User by name or id number, or putting a User object back into the datastore.

If you're using Eclipse, it will complain about the lack of a UserServiceAsync interface, and will build it for you automatically if you let it. It's basically the same as above, but doesn't need to extend RemoteService or declare a @RemoteServiceRelativePath. You can grab the complete code from github, as shown above.

In the server package, we need a servlet class that implements the UserService interface we just defined.

com/lisedex/volinfoman/server/UserServiceImpl.java
public class UserServiceImpl extends RemoteServiceServlet implements
                UserService {

    private DAO dao = new DAO();

    public User getUser(String username) {
            return dao.getUser(username);
    }
}

The real code has stub implementations for getUser(Long id) and putUser(User user) since we're not using them for now. We want to abstract away how we'll be accessing the datastore in case we need to change it, so we put actual interaction in a Data Access Object called...DAO.

com/lisedex/volinfoman/server/DAO.java
public class DAO extends DAOBase {
        static {
                ObjectifyService.register(User.class);
        }

        private Objectify ofy;

        public DAO() {
                ofy = ObjectifyService.begin();
        }

We don't have to extend the DAOBase class, but it gives us a couple of functions that we can use without coding, such as a lazily instantiated Objectify object. Of course, I didn't know that at the time I wrote the code, so I created my own. Oops.

You need to register the classes you'll be using with Objectify, and calling ObjectifyService.begin() returns an Objectify object we'll use to interact with the datastore.

public User getUser(String username) {
                User fetched = ofy.query(User.class)
                    .filter("username", username).get();
                return fetched;
        }

        public User getOrCreateUser(String username) {
                User fetched = ofy.query(User.class)
                    .filter("username", username).get();
                if (fetched == null) {
                        fetched = new User(null, username, 
                            User.STATUS_INVALID, null, null, null, 
                            null);
                        ofy.put(fetched);
                }
                return fetched;
        }

        public void putUser(User user) {
                ofy.put(user);
        }
}

To get a User by username, we need to execute a query on the datastore for all User objects, and filter the results by the username field. If no such object is in the datastore, the get() method will return a null.

Writing the object is easier, since we already know the Key for the object, as it's encoded in the class (User.class, and Long id). Just pass the object to Objectify's put() method. If the id field in the object is of type Long, and its value when passed to put() is null, Objectify will automatically generate an id for the object. If the id is of type long or String, the developer is responsible for that themselves.

The getOrCreateUser() function should be modified to use the get() and put() methods in DAO. Another oops. And I need to add more data checking, such as if the User sent to put() is null.

Make the Login button do something


Now we need to wire this stuff into the user interface, so that when we hit the Login button, we actually go out to the datastore and try to retrieve the appropriate User.

com/lisedex/volinfoman/client/DefaultHomepage.java
class MyHandler implements ClickHandler {

    @Override
    public void onClick(ClickEvent event) {
        sendStatus.setText("Sending..." + username.getText() +
                           "/" + password.getText());
        sendButton.setEnabled(false);
        userService.getUser(username.getText(),
            new AsyncCallback<User>() {
                  public void onFailure(Throwable caught) {
                      sendStatus.setText(sendStatus.getText()
                                 + "   FAILED");
                      sendButton.setEnabled(true);
                  }

                  @Override
                  public void onSuccess(User result) {
                      if (result == null) {
                             sendStatus.setText(sendStatus.getText() 
                                 + "    NO SUCH USER");
                             sendButton.setEnabled(true);
                             return;
                      }
                      sendStatus.setText(sendStatus.getText()
                          + "    SUCCESS: " + result.toString());
                      sendButton.setEnabled(true);
                  }
              });
    }
}

sendButton.addClickHandler(new MyHandler());

We obviously have not implemented the MVP model, as all of this stuff is in our view. In the ClickHandler inner class, we call the UserService.getUser() asynchronous method with an AsyncCallback inner class that will handle what happens when the RPC call returns.

If the RPC call fails, onFailure gets called with an exception. This doesn't happen if the username doesn't exist in the User indexes; it only happens with the client is not able to access the servlet for some reason. onSuccess is called with a User object when the RPC call works. The result is null if no such User was found, otherwise it's populated with all stored fields for the object.

Next up: Guice, security


Guice is the basis for Gin, except Guice provides dependency injection for Java generally, where Gin is for GWT. DAO's are a perfect place to apply dependency injection, since we may want to use the real datastore back end, or a dummy used for testing. We'll cover that in the next installment.

Also, there's no security or authorization needed to call any of the RPC functions. Anyone that wants can read or write any User object to my datastore, even if they're not using my client application. We'll, uh, get to that sometime.

Sunday, July 18, 2010

Dependency injection and Gin

I was coming across the phrase "dependency injection" quite a bit while reading articles about GWT and MVP, and references to the Guice and Gin libraries coming out of Google. I figured it was some new programming paradigm and that I was hopelessly out of date not knowing what it was. The wikipedia entry didn't really help clear it up, nor did other articles I read. I kept getting the feeling that I was missing something, and that the concept wasn't really something new.

Turns out I was missing something, all right


I was missing that DI is giving a name to something that I and every other object oriented developer have been doing for years.

The short summary in this great article is what finally made me realize that I could stop trying to make it more complicated than it was: "Dependency injection means giving an object its instance variables. Really. That's it." If you're having trouble with the concept, James Shore's article is invaluable.

A conceptual example would be something like: you have an object that relies on a reference to some sort of data provider. In production, the provider will be a database back end, but during testing you want to mock up a provider that is always up and you can precisely control what errors it has.

You might solve this by constructing a factory that spits out the correct type of provider, and your object can ask that factory to give it a reference to the proper kind of provider. This works, but your testing has to construct, tear down, and reset the factory to the previous state between each test. You also have to write a lot of boilerplate code for each of these factories.

DI refers to giving the object a reference to the proper provider, which, during testing, allows you to build a new provider instance which you pass to the object during each test. As you exit the test, the object and provider are dereferenced automatically, saving you the cleanup. But in the code itself, you're now either passing provider references possibly through layer after layer of constructors, or you're building factories for the initial object which get the proper provider and pass it to the object, and then return the completed object.

Guice and Gin save you from all this repetitive factory building and wiring, and what provider each object needs can be declared in one place, or inline where the instantiated providers will be injected in the object.

I'm certainly not qualified at this time to discuss all of the nuances and capabilities of these libraries, since there are many different ways this concept could be used.

Instead, I'll talk about using Gin in this project


com/lisedex/volinfoman/Volinfoman.gwt.xml
<!-- Include Gin -->
<inherits name="com.google.gwt.inject.Inject" />
Adding those lines to your module's gwt.xml file, and adding the Gin and Guice jar files to your classpath, is all that it takes to enable the Gin annotations and compile time code generation.

If you go to the VolInfoMan project page on github, you can go to the basic-gin tag and download the source. The project is very, very basic at this point, and it's easy to find the one instance of using Gin.

As practice, I wanted to set up the home page you land on when loading the site to be capable of being changed by dependency injection. This will allow, later, changing the implementation of the page by making a new class that extends the skeleton Homepage class, and just changing the Gin binding in one place, and all references to that Homepage will be using my new implementation. Look ma, no factory!

com/lisedex/volinfoman/client/Homepage.java
package com.lisedex.volinfoman.client;

import com.google.gwt.user.client.ui.Composite;

public class Homepage extends Composite {
}

First we define our skeleton Homepage class, which will be added to our RootPanel later in onModuleLoad() (actually onModuleLoad2() due to our use of the gwt-log library).

com/lisedex/volinfoman/client/gin/VolinfomanGinjector.java
package com.lisedex.volinfoman.client.gin;

import com.google.gwt.inject.client.GinModules;
import com.google.gwt.inject.client.Ginjector;
import com.lisedex.volinfoman.client.Homepage;

@GinModules(VolinfomanModule.class)
public interface VolinfomanGinjector extends Ginjector {
 Homepage getHomepage();
}

We extend the com.google.gwt.inject.client.Ginjector class and use the @GinModules annotation to specify what class contains the binding configuration. We also define the getHomepage() injector method, which we need when using Gin in our application initialization code, because Gin is working at compile time to generate JavaScript, unlike Guice. Other dependencies that operate below our initialization code will be injected automatically, and won't require this type of method.

com/lisedex/volinfoman/client/gin/VolinfomanModule.java
package com.lisedex.volinfoman.client.gin;

import com.google.gwt.inject.client.AbstractGinModule;
import com.lisedex.volinfoman.client.DefaultHomepage;
import com.lisedex.volinfoman.client.Homepage;

public class VolinfomanModule extends AbstractGinModule {
        @Override
        protected void configure() {
                bind(Homepage.class).to(DefaultHomepage.class);
        }
}

We extend AbstractGinModule to declare our bindings for injection. Using AbstractGinModule instead of GinModule allows us to drop the binder.bind() syntax in favor of bind(). In the above code, we're declaring that when our code asks for a Homepage object, it will get a DefaultHomepage instead.

com.lisedex.volinfoman.client.DefaultHomepage actually contains no references to Gin. It extends Homepage but doesn't need to know anything about injection.

com/lisedex/volinfoman/client/Volinfoman.java
package com.lisedex.volinfoman.client;

import com.lisedex.volinfoman.client.gin.VolinfomanGinjector;
// import GWT objects here...

public class Volinfoman implements EntryPoint {
        public void onModuleLoad() {
                // we set up gwt-log here and
                // use a DeferredCommand to execute
                // onModuleLoad2() */
                // ...
        }

        private void onModuleLoad2() {
                // Create a ginjector
                VolinfomanGinjector ginjector = 
                    GWT.create(VolinfomanGinjector.class);
                // Add the homepage to the rootpanel
                RootPanel.get().add(ginjector.getHomepage());
        }
}

And our GWT entry point creates the Ginjector, and we use the injector method getHomepage() to inject an instance of DefaultHomepage, thanks to the binding we set up in VolinfomanModule. As mentioned above, we have to use an injector method due to current limitations in Gin.

That's it. Remember to grab the basic-gin tag of the source code from github for a compile-ready Eclipse project where you can immediately play with Gin.

Other Gin information


Gin supports some of the same annotations as Guice, such as @Inject, @ProvidedBy, @Singleton, @ImplementedBy, all of which allow you to move much of the binding information out to the source where the bindings occur. Much of this is to provide an alternative to using a GinModule to declare all of the bindings in one place; the latter is the option I prefer, but it's personal preference for the most part.

Some of the other bind() syntaxes are:
bind(Something.class).toProvider(SomethingProvider.class);
bind(Something.class).to(SomethingImplementation.class)
    .in(Elsewhere.class);
bind(Something.class).annotatedWith(SomethingAnnotation.class);
    .to(SomethingImplementation.class);
bind(Something.class).to(SomethingImplementation.class)
    .in(Singleton.class);

Line 1: When Gin injects an instance of a class normally, it uses the class's default constructor. If you need to give extra information to a constructor, you can use the toProvider() syntax. Gin will inject an instance of Something from the SomethingProvider wherever you need a Something class.

Line 2: If you only want a binding to apply in Elsewhere, instead of globally, you can use this syntax to specify that scope.

Line 4: You can also set up custom annotations. When you want multiple bindings for one type, you can specify which binding to use by setting up an annotation for each. The annotation and the type will uniquely identify which binding is appropriate. See the Guice wiki for more information.

Line 6: Gin's injector normally creates a new instance of each SomethingImplementation when it gets injected. However, sometimes you'll want to use the Singleton pattern, and only have one instance of SomethingImplementation running around. Remember that this is a Singleton per Ginjector, so if you have multiple Ginjectors (which, if you do, you probably need to double check your code) there will be one instance of the SomethingImplementation for each.

There's a bunch more that can be done with Gin; this is obviously only scratching the surface. Remember that Gin does the injection at compile-time, so there should be almost no overhead for using it, and it can save you a lot of repetitive coding. To understand all of the capabilities, it would behoove you to read the Guice User's Guide, the Gin tutorial, and the compatibility information between Gin and Guice. The documentation for Guice is much more thorough, but you have to know which ideas can be used in your GWT code. Guice will be used later for our server side code which runs under GAE.

Monday, July 12, 2010

More research...by watching YouTube videos

I'm itching to start hacking code, and I probably will this evening, but I forced myself to spend a good bit of time yesterday doing research: I read articles at the Google App Engine site, GWT tutorials, and best practices for both GAE and GWT. Generally, what I got from those is how much I have to learn about these technologies, which is a little depressing. Also, that if I'm not really focused, I'm going to get sidetracked by reading about issues I'm not even facing yet. But the up side is that there's so much information out there, already available, that most of the problems I'll run in to someone else has already hit and written about it.

Google IO videos, or how I learned to stop multitasking and love YouTube


The Google IO talks, presented on the Google Developer's Channel on YouTube, were probably some of the most informative stuff I looked at yesterday, as well as exemplifying my tendency to get sidetracked; do I really need to watch an hour and a half of the Google Wave product launch? I'm not even going to provide a link to that. It also served to remind me that sometimes trying to multitask is inefficient, as when I'd put the videos in the corner of my screen and try to do other work, I'd completely miss huge chunks of the talk. Having to continuously rewind, and, while the video was getting back to the section I missed, going back to my regular work, only to completely miss the same section again...well, it started to border on the absurd. Biting the bullet, and telling myself that just watching the video with no distractions was actually most the efficient way to get this done, was also the way to get the most out of these videos. And there's a ton of good information there.

Max Ross easily rose to the top as one of my favorite speakers. During his videos, I would actually neglect my work and focus completely on the video. The first talk of his I watched was an explanation of schemas in the data store, since I'm not going to have complete SQL functionality while writing an application for the GAE. I also listened to his (audio only) talk about Hibernate Shards, even though I won't be needing that, most likely ever, for this particular project. Speaking to Max's skill as a speaker, my girlfriend, who actively tries not to learn about computers almost to the point of Luddism, got sucked in while I was listening. I believe that, initially, it was her mishearing the word "sharding" as "sharting," and that hooked her scatalogical mind. But she ended up putting down her book, and listening with me. By the end, she actually asked me questions that showed she grasped a lot of what Max was talking about. She's a smart girl, but I was pretty surprised.

Testing


I'm a big proponent of proper testing when I'm developing software, and though I occasionally get some push back for that, it really helps my confidence in code as a project progresses. So it was nice to hear from Max that he really believes in testing, too, during his talk about GAE Testing Techniques. Using the cloud as a test farm is pretty ingenious, so I wastedspent some time looking at Cloud Cover test harness. The code hasn't been touched since May, but I really like the idea of testing my GAE code actually on the App Engine servers. This led to GWT testing, which reinforces the idea that the MVP model will promote not just a clean conceptual layout for the project or make it easier to implement code splitting, but will also make it much easier to have GWT test cases that will run very fast, except for the small percentage that need to test the GUI widgets themselves. This talk also pointed me to WebDriver, which is the heir apparent to Selenium. The fact that Selenium ran the testing through JavaScript always seemed a bit too abstracted from the browser itself to provide true test from the user all the way to the data store. WebDriver will plug directly in to the browser and generate actual key strokes, button clicks, and other events.

Odds and ends


I also watched a talk about some of the new stuff in Google App Engine (at least new since the 2009 Google IO conference), developing complex and scalable apps in GAE, and new query capabilities that have been added to the data store.

And I believe watching all of that in one day kind of made me a bit loopy, and I know that a lot of the specifics ended up getting pushed out by new specifics; but now I have a better idea of what's out there, so when I do run into an issue or question, I know where to look. This post is actually more for me, so I have a place to refer with all of these links. I also have a good idea of how to implement my testing, and how the project should be designed initially.

It was actually time well spent, even though I had to suppress my urge to just start coding, and I'm now in a better position to start without having to feel like I'm going to have to revisit, in the near future, every single decision I make.

And I need to get over my mancrush on Max Ross. He's so dreamy.

Sunday, July 11, 2010

A short note on Cygwin git

I use git for my source control, and when developing under Windows, I use the Cygwin environment for all of my command line needs.  My background is primarily Unix, so I'm used to having a powerful shell prompt and scripting for a lot of tasks.  I also use the git available from Cygwin instead of the one from the git site.

You will quickly run into execute permission problems and CRLF vs. LF translation problems.

Execute bit


Windows doesn't manage the execute bit very well, especially on a FAT filesystem, but I find it even on NTFS.  Files created under Cygwin will be created according to your umask, usually as mode 0644, while files from Windows will show up as 0700.  This isn't a real problem except that git tracks the mode of each file in the repository, so changes to the mode will show up as the file being modified.  Setting core.filemode to false tells git to ignore the executable bit.  If something slips in, you may have to use git update-index --chmod=-x to correct it.

core.filemode is set in your .git/config file by git init regardless of how it's set in your system or global configuration, so you will need to set it in your repository configuration.

CRLF vs. LF


Files created under Cygwin will use only LF to mark end of line, while files created under Windows applications will use CRLF.  By setting core.autocrlf to true, Cygwin git will convert all CRLFs to LFs on commit operations, and all LFs to CRLFs when checking out.

I also set core.safecrlf to warn, which checks, before adding a file, that doing the crlf conversions is "safe".  We want to make sure that we'll get the same file back when it's checked out.  Git cannot "safely" convert files that contain both LFs and CRLFs, so a file that's been edited under both Windows and Cygwin may produce warning: LF will be replaced by CRLF in Foo.java when running git add.  This is not a problem with text files, because the file will be cleaned up during its next commit so that all EOLs are the same.  However, binary files will be corrupted if they are not properly detected by git.  I have never seen git label a binary file as text, but if this does occur you can set the file in .gitattributes as binary.  For example
   *.jpg binary
will make git consider all .jpg files as binary, and will not apply the CRLF conversion, nor will it try a text diff.  

You can also set core.safecrlf to true instead of warn, in which case the error will be fatal, and you will have to correct the problem before you can add the file.  You can run a text file through unix2dos, or add the binary file to .gitattributes, but since git bats a thousand, in my experience, at detecting binary files, it's a waste of time to have to manually fix all of your text files.

TL;DR


So in conclusion, if developing using git under Cygwin, I run the following commands after creating a repository:
   git config core.autocrlf true
   git config core.safecrlf warn
   git config core.filemode false
Explanations of these settings can be found on the git-config man page.