« March 2008 | Main | May 2008 »

April 2008

Apr 30, 2008

H2 Database - Performance Tracing

I first wrote about the H2 Database last August. I've been using it since and so far I'm very happy with it. The article I wrote is still getting hits from people searching for information on H2 (here's a fun fact, if you search Google for H2, you'll get the H2 Database before the H2 Hummer) . I'm now working on improving the performance of my application. Like everything else in H2, it is very simple to trace the database performance. This article demonstrates several techniques.

Note: the tools shown here are relevant to H2 starting from version 1.0.70. Since H2 is still work in progress, this may change in the future. I'll do my best to keep this article up to date. I also included some useful links to the original H2 Tutorial.

Generating Trace Files

H2 can output trace files which include the performance information. These trace files contain all the commands executed and their execution details. In the examples hereunder, I will use a database called mydatabase.

First, you need to specify the trace level.  You do this by setting a property called TRACE_LEVEL_FILE. The best way is to simply specify this upon connection to the database. This is done by adding ";TRACE_LEVEL_FILE=n", where n is the trace level, at the end of your database connection string.  The trace levels are: 0=off, 1=error, 2=info, 3=debug. Use trace level 2 or 3 in order to get the performance information. For example:

jdbc:h2:file:~/mydatabase;TRACE_LEVEL_FILE=3

From that point, H2 will start writing trace logs to the directory where the database file is located. The trace file is called mydatabase.trace.db. Once the file reaches the maximum size, it is rotated to mydatabase.trace.db.old (the old one is erased). You can set the size of the log file by setting TRACE_MAX_FILE_SIZE. My connection URL, looks like this:

jdbc:h2:file:~/mydatabase;TRACE_LEVEL_FILE=3;TRACE_MAX_FILE_SIZE=20

H2 is very tolerant regarding trace files. You can rename the file and it will create a new one instead. You can set the trace level at any given time, not only upon connection. You can use the H2 console for that, just connect to your database instance and execute the command:

SET TRACE_LEVEL_FILE=3

Using the Trace Files

SQL commands in the trace files will have the following format:

04-30 11:47:14 jdbc[10]:
/*SQL l:61 #:14 t:1*/SELECT * FROM EMPLOYEES WHERE EMP_ID = ? {1: 243};

The first line shows the date and time. The second line shows the following information:

  1. l:nn - The length of the SQL. Will not appear if there are no parameters (see below).
  2. #:nn - The number of rows returned or affected. Will not appear if no rows were found or affected.
  3. t:nn - The execution time in milliseconds. Will not appear if the time was less than 1 ms.
  4. The SQL command itself with all the parameters, in case you're using a prepared statement. You can simply copy the SQL with the parameters and execute it through the H2 Console, which is very handy for debugging.

The trace file is your first stop when checking your performance. You can check every single command and see how it performs.

SQL Command Statistics

H2 includes several tool, among them a utility that generates statistics from a trace file. The tool is called ConvertTraceFile. The tool takes a trace file as input and has two outputs: a Java class (not relevant) and an SQL script which contains all the SQL commands and the execution statistics.

The ConvertTraceFile is invoked in the following manner:

java -cp h2.jar org.h2.tools.ConvertTraceFile -traceFile mydatabase.trace.db

You should specify full paths for h2.jar and your trace file if necessary. Alternatively, make sure h2.jar is in your CLASSPATH. The SQL script will be in a file called mydatabase.sql. You may also specify a different file name (use -? to see all the available options). 

The SQL script file holds all the SQL commands which were executed when the trace file was active. At the end of the file, you will find the command statistics section which has the following form:

-- self accu time count result sql
-- 89% 89% 3309    12     234 SELECT * FROM EMPLOYEES WHERE EMP_ID = ?;
--   5% 94% 192 1015   1015 UPDATE EMPLOYEES SET NAME=?, DEPT=?, MANAGER=? WHERE EMP_ID=?;
--   2% 97%   87   421    421 UPDATE DEPARTMENTS SET NAME=? WHERE ID=?;
--   0% 97%   26 8123   8123 CALL AUTOCOMMIT();
--   0% 98%   16    18    736 SELECT * FROM DEPARTMENTS WHERE ID = ?;
--   0% 98%   14   424    740 SELECT COUNT(*) FROM DEPARTMENTS;

The columns are:

  1. Self - The percentage of the given command's time from the total time.
  2. Accu - The percentage of the accumulated time from the total time. The accumulated time is the time of the commands listed so far.
  3. Time - The total time of the command executions in milliseconds.
  4. Count - the number of times the command was executed.
  5. Result - the total number of fetched or affected rows.
  6. SQL - the command itself

Needless to say, the statistics you get can easily be imported to OpenOffice Calc or Micsoroft Excel for further manipulation (you can use a fixed column import).

Conclusion

I like simplicity and this is one of the main reasons I enjoy using H2. The above tools enable you to get statistics and detailed information on SQL queries and commands executed in H2.

Good luck with your performance tuning.

Apr 24, 2008

You've gotta love Platform Independence

It has been a rough week. Last week my MacBook Pro burned his logic board and then I managed to brick my PC. This week, though, I got a new mother board (with a decent Core 2 Duo processor) for my PC and I'm up and running.

Going back to Windows is annoying as expected. The worst annoyance comes from the difference  in the location of the Mac "Command" and the Windows "Ctrl", especially when it comes to copy/paste: in OS X it's Command-X and in Windows it's Ctrl-X, but the Command is placed where the Alt key is in Windows. I also miss Quicksilver and Exposé. I know, there are alternatives, but I just couldn't be bothered to install any.

On the upside, my environment is surprisingly the same. Here's some examples:

  • Eclipse - exactly the same. Right down to the plug-ins. The keyboard shortcuts are a bit different which takes some time to get used to.
  • Subversion - I just got my repository from my Mozy backup, installed the Subclipse Eclipse plug-in and that's it.
  • H2 database - completely cross platform. I didn't even bother to install it, just downloaded the generic package and unzipped it. Works beautifully and exactly the same (maybe faster?).
  • Firefox - yes, I have all my add-ons running. It's feels at home.
  • Open Office - I had to work on some documents. On the Mac I use NeoOffice. My very complex spreadsheet didn't even notice the platform change.
  • I use Cygwin which gives a Unix prompt, much like the one I get in OS X. Cool.

After some minor tweaking the application I'm developing was up and running. It was working beautifully. I was working fluently and almost forgot I switched the platform. I made some progress this week, but...

  • All my tasks are managed in OmniFocus - I couldn't find anything that can open my backup files on PC.
  • All my brainstorming and designs are in OmniOutliner format - again, I couldn't read it. Not even close.
  • I had some notes in xPad. I managed to partially recover the ones I needed.

Some after thoughts after this "fun" week:

  • The Mozy backup service allows downloading backups through a web clients. However, Mac backups are downloaded as DMG files. It took me time to open these on Windows. I finally used IsoBuster, which isn't free. I'm just wondering what's wrong with plain-old ZIP files.
  • I have backups of my sources and documents. I missed out on some important stuff - like Eclipse workspace settings (including launch configurations and what not), my test projects, etc. Luckily, I'm going to get my MacBook Pro early next week with the hard-disk intact.
  • It took me hours to install the Eclipse & Firefox with all the plug-ins and settings. This gave me a great idea - write a plug-in which synchronizes the settings and plug-ins. You could then download a distribution of Eclipse/Firefox with this plug-in and the plug-in will install your other plug-ins and do all the configuration for you. Sounds like a million dollar idea to me.

Finally, this is the price of going solo and not having an IT department behind you. I knew that when I "signed up". This helped reduce the anger and pain involved. I knew this is going to happen some day and I just said to myself: "well, today is the day".

Apr 21, 2008

Off the Grid

Last week, my MacBook Pro died on me, as I wrote in my previous post. I started working on my PC, which is a very old and slow machine (a Pentium 4).  After a couple of hours I realized that it won't  work without upgrading my machine's memory. However, I couldn't upgrade the memory since my machine would not accept 1 GB memory modules. They told me it has something to do with the BIOS, which needs an update. So, I got the BIOS updating software, updated my BIOS and, as you might expect, ended up bricking my PC. Later that day I got a word from the lab that my MacBook Pro has toasted its' logic board and they're sending in a new one (which will take time).

Two machines. Two totaled logic boards. Two days. I'm off the grid.

I have no problem being disconnected from the net. I usually do that when I take a vacation: no internet, no e-mail, minimal phone calls. However, sitting at home with no computer was a strange feeling. Luckily, I could browse my Gmail from my 3G cell phone, although this wasn't very satisfying. Now, I got a spare machine from my brother and I'll do my best not to trash it. I should be getting my PC back today with a revamped processor.

Happy Passover everybody.

Apr 17, 2008

I'm switching after all...

In an unfortunate turn of events, my MacBook Pro has decided to take a vacation. It's spring time, so there must be something in the air. Maybe it's the upcoming Passover, which is known to be "the festival of freedom". Anyway, it just died on me. The Apple service in Israel has room for improvement: they promise to fix it in two weeks. Two weeks. What am I going to do in the mean while?

I wasn't really shocked when it happened. My personal experience tells me that laptops, no matter how good they are, will need service every year or so. It's probably because all the miniature and delicate components cramped in this overheating space, which constantly gets rattled and banged. My previous laptop was an IBM ThinkPad T41. It was a good and sturdy machine, but I did visit the IT department every now and then. The advantage of working in a big company is that I got a replacement laptop (with my hard drive) whenever such a mishap occurred, so I could keep on working. Now, I'm decommissioned.

I'm making the best of this time to finally test my product on Windows. It took me a few hours to restore my environment. I backed up my source repository, but not much else, which means I have to restore my workspace. I had to download all the relevant products (Eclipse, Spring, H2, ...) and install them. I also had to work around issues of platform compatibility, mainly resource locations. However, once all this was done, I was up and running.

I appreciate the platform compatibility, but It's not perfect. The Eclipse UI is based on SWT. SWT is much better looking than Swing since it is wrapping native UI controls. Which means it may behave a bit different under each platform. It is cross-platform, but it is far from being bug free and requires the developer to test and debug the product on all platforms. My product is running, but not without glitches. Well, I have the upcoming two weeks to get rid of those. And maybe I'll get a little freedom myself.

Apr 14, 2008

Who Needs an Online IDE?

[Note: I wrote this article for EclipseZone.]

We've seen some hype lately around the next Eclipse release dubbed E4. The theme of E4 would be to take Eclipse online. This is not the first attempt (I wrote about it 6 months ago), however, now it turned from a sporadic effort to a full blown platform migration.

The Eclipse Duality

Before looking deeper into this trend, there's a much needed distinction to be made. Eclipse, as a brand, stands for two different entities: the Eclipse IDE (Integrated Development Environment) and the Eclipse RCP (Rich Client Platform). It's easy to get confused, but it's not the same thing. It's almost like the difference between Microsoft VisualStudio and the .Net platform.

The Eclipse IDE is a great tool for software development. You can use it to develop in Java, C++, Ruby and many other languages. On the other hand, the Eclipse Platform is a framework for developing Java based cross-platform desktop application. The Eclipse IDE uses the Eclipse Platform as its' base. The Eclipse IDE is the best tool for developing applications for the Eclipse platform, however, one may use any IDE which supports Java development for that purpose.

In this post, I will focus on the Eclipse IDE, looking forward to E4.

Who Needs an Online IDE?

The trend of taking desktop applications online and providing them as a service (SaaS) is not new. It was only a matter of time until it gets to the IDEs. It's taking longer for the IDE market to be affected, probably because it is a much smaller market than, let's say, word processors. The online word processors market is already saturated with products like Google Docs and Zoho Writer.

Zoho-creator-user-defined-functions

Most online IDEs are very specific (unlike generic IDEs like Eclipse), geared towards a given service or platform. The IDE is not the service being provided, but rather a tool for the user to consume other services. For example, the service can be an online database and the online IDE would allow developing solutions on top of this service: database definition, screen design, coding and even debugging. The deployment is seamless, using the provided service.

Examples for such services are Coghead, Zoho Creator, Bungee Builder, Microsoft PopFly and Yahoo Pipes. These are all proprietary services. Some use their own languages and all of these services are restricting the deployment to their servers. This is usually their underlying business model: providing the service for executing the complete application.


Heroku-code-editor

There are a few services which are more generic by nature, based on standard languages. For example, Heroku is using Ruby on Rails (RoR) and can be used develop and deploy a full RoR application. A unique feature in Heroku is the ability to export your code. Since the code is generic RoR, you can then deploy it on any server that supports RoR. Heroku is based on Amazon EC2 platform. I won't be surprised to see Python-oriented services based on Google's AppEngine soon. Another interesting example in this area is AppJet which uses JavaScript.

This online IDE roundup cannot be complete without mentioning CodeIDE and ECCO. Both provide completely generic online IDEs. They seem like perfect tools for people learning how to program, allowing the user to instantly start coding and see the results. Although these are not complete IDEs, they provide a glimpse of what a generic online IDE might look like.

All the services above have mainly two thing in common: they do not require any installation and moving from coding to execution is just a few clicks away. Usually, there are considerable overheads in IDE installation, environment configuration and server deployment and execution. This can all be avoided, saving a great deal of time (and agony, speaking from personal experience).

Nevertheless, none of these services is even close to the rich feature set you get from a true desktop IDE like Eclipse or VisualStudio. This gap is not unique to IDEs, though. People switching from Microsoft Word (or Apple Pages) to Google Docs will feel the difference. Sure, Google Docs has all the basic features. However, it is still very far from a full-fledged desktop word processor.

You can expect this to change. The following extract taken from an article in CIO magazine, quoting Gartner analyst Neil MacDonald:

The move to server-agnostic applications is still in its infancy but will soon have a major effect on enterprise computing. The legacy applications won't go away, even if the exciting stuff is being done on Internet-based apps, they said. But it won't stay that way. Today, 70 percent to 80 percent of corporate applications require Windows to run, but the Gartner analysts expect a tipping point in 2011, when the majority of these applications will be OS-agnostic, such as Web applications.

The Eclipse IDE is just joining the crowd. We're going to see an increasingly large number of online applications replacing their desktop counterparts. IDEs that won't offer an "online option" will be falling behind.

The Future

As I mentioned, there are some immediate benefits from having an online IDE, especially: no installation and instant deployment. IMHO, this is just the tip of the iceberg. Here are just a few thoughts of what we might expect:

     
  • Online IDEs open new capabilities of sharing and collaboration. Consider doing pair-programming with your colleague, which is sitting in another continent.
  •  
  • This can be even more useful when it comes to outsourcing and your coworker is an occasional developer. For example, I need the services of an expert DBA. I can find one online in elance and work together on my project immediately.
  •  
  • On-demand services open new possibilities: instead of buying a profiler for $500, maybe I'll just pay for the time which I'm using it.   
  •  
  • Mash-ups: I can use Google Page Creator to design my project's web pages and Yahoo Pipes to define a web service. All as part of the same project. Today, Eclipse is already a mash-up of OSGi services. Tomorrow, it may be a mash-up of web services or REST.
  •  
  • All the online services I mentioned in this post had to developed their own IDEs. I'm sure future vendors will be more than happy to use an existing online IDE and develop plug-ins on top of it. As the number of online services provided grow, the need becomes more imminent.

I'm just guessing here, but it does open up new directions and opportunities.


Apr 02, 2008

No, I haven't switched back...

Yesterday's post, obviously, was an April fools' joke. I'm not switching back to Windows and there is no software which sends you electrical shocks through the keyboard.

Some after thoughts:

  • It's an interesting marketing technique: it may be easier to get bloggers to mention your site free as part of a joke. If you're smart enough (and have a good sense of humor), you can even generate some hype out of it.
  • There is a connection between managing my tasks and managing my machine time. So, why am I using two separate tools for that, that has no integration between them?  There's a correlation between the RescueTime Tags and the GTD contexts. Just a thought.

Apr 01, 2008

The software that made me switch back to windows

[This was an April Fool's joke. See the next post.]

I didn't think that this day will come. Well, I was wrong. I came across this amazingly innovative piece of software which completely changed the way I use my machine. It's called FruitfulTime TaskManager Electro. Essentially, it's a task management software which also keeps track of what you do on your machine accordingly. I had two separate systems for that on my Mac: OmniFocus for task management and RescueTime for logging my machine usage.

Taskmanager_electro_box However, there's a big difference. FTE can help you prevent procrastination by giving you a small "reminder" whenever you start doing unrelated stuff on your machine. Now, you're probably thinking: "I can ignore those remainders, there's nothing new here". Trust me, you won't ignore this one, since it comes in the form of an electric shock.

I took part in a private beta over the past 3 weeks and I can just tell you this: amazing. My productivity went up by roughly 70%. As I said, it only works on Windows. A Mac OS X version is not planned at the moment, so I had to switch to Vista on my Mac. Well, my MacBook Pro was declared in 2007 as the fastest laptop for running Vista by PC World, so no worries there. My only concern was that it drained the batteries of my Wireless Mighty Mouse. Nevertheless, the FruitfulTime team assured me this will be solved until the beta is complete.