My Struggle

How to make Snow Leopard REALLY work with Exchange

2009-10-26T10:56:00.002Z

I was very excited when Apple released the Snow Leopard, also known as Mac OS X 10.6. The main benefit would be the native support of Microsoft Exchange in iCal, Address Book and Mail.app. And this is something I was complaining about for years.

So, I upgraded my computers as soon as I could, and I was initially quite surprised how easy it was to configure the applications to work with Exchange server. (Yes, we have Mictosoft Exchange 2007 over here).

But the very next moment some dire disappointments came:

Push email doesn't work with Exchange account any more. If I want to be able to react to emails instantly, in a kind of IM way, I would have to poll the servers every minute or so. This clogs the network and puts extra workload on my workstation. And it is so archaic to poll your email in 2009.

Every so often the message counter would show that I have some unread messages, while in fact I don't. White day glitch. I would have to go to "Mailbox ⟶ Rebuild" to reindex the account to get rid of this glitch. And it would happen again next day.

After suffering for a couple of weeks and hoping for a fix from Apple, I have found a workaround to make Snow Leopard work with Exchange almost the way I want it to work. The whole procedure is very simple and only takes 10 seconds more than usual setup. The only additional requirement is to have IMAP access enabled on your Exchange server. Ask your system administrator.

Open Mail.app and add your Exchange account just the way you normally would. This will add the Exchange account to your iCal and Address Book as well.

Now, you have to remove the newly created Exchange account from Mail.app. This would still leave the accounts for iCal and Address Book.

Create a new account in Mail.app using "Exchange IMAP" account type instead. Sometimes this option is not available as Mail.app detects your Exchange server and disables the choice. Just type a fake email address in the first screen to avoid that. Something like asdfhj@sad.geweg.aj.net would do. Then select "Exchange IMAP" as an account type, and fill in all the appropriate options. Don't forget to change your email address to a normal form (from a fake one) once you have your account created.

Enjoy push email, proper counting of unread messages, and iCal support of Exchange calendars at the same time!

iPhone support in Google Calendar

2009-02-10T15:13:00.001Z

Hurray! Now google calendar supports instant sync with an iPhone!
http://www.google.com/mobile/apple/sync.html

Google Calendar now supports Apple iCal

2008-12-03T17:44:00.002Z

Breaking news from Google: Google Calendar now supports Apple iCal.

It is not really a solution for synchronising with Exchange, though a very useful extension to Google calendar. With a couple of tweaks it will be possible to synch Google calendar to Exchange too. The only feature missing that prevents Google calendar to be a proxy between Exchange and iCal is free/busy service. I am afraid that the only solution for fellow Mac users is to wait until Snow Leopard is released, and hope that the new incarnation of iCal would support Exchange natively.

Follow the link to the official Google blog above to find the instructions on how to use Google calendar with iCal.

Exchange support coming to iCal next year

2008-06-17T15:06:00.002+01:00

Finally some good news. It is official now, iCal will support Microsoft Exchange calendars in the next version of Mac OS X which is expected to be released in 2009.
The official Apple press release can be found at http://www.apple.com/pr/library/2008/06/09snowleopard.html and it states:

For the first time, OS X includes native support for Microsoft Exchange 2007 in OS X applications Mail, iCal® and Address Book, making it even easier to integrate Macs into organizations of any size.

The only problem is that we have to wait for at least another year.

Will Mobile Me solve the synchronization problem?

2008-06-09T20:47:00.002+01:00

Apple have announced an extension to their .Mac service, and they have rebranded it as "Mobile Me". The initial information available at their web pages at http://www.apple.com/uk/mobileme promises seamless integration with Mac and PC. They particularly claim that Mobile Me calendar will synchronize with Outlook and iCal and will provide push updates across all the computers connected to the account.

Imagine the following scenario: you run Microsoft Outlook on Parallels or on a dedicated computer at work. Outlook synchronizes your Exchange calendar to Mobile Me, and your iCal also works through that service.
Given that Mobile Me is developed by Apple, I'd imagine that it would actually work and be more secure than using some third party synchronization proxies from companies I have never heard about.

Unfortunately, they are not rolling Mobile Me just yet, but please come back after they have, and I'll tell you if it worked for me.

Any speculations on whether it is going to work or not?

Any hopes for Exchange support now?

2008-03-06T18:57:00.002Z

I am currently watching the iPhone SDK press conference. They have announced that the iPhone will soon support proper communication with the Exchange. This includes complete Calendar support. What I wish they have done is to license the same technology for the iCal. That would have been an ideal solution for all of us who suffers being a Mac user within a corporate environment.

I do understand that it may be possible to sync the appointments between iCal and Exchange either by using current solutions or through syncing with an iPhone, but what I am looking for is free/busy integration as well.

So, we all have two hopes now, the first one is that the Groupcal will be updated, and the second one (Oh, that would be fantastic) that Apple will begin to support enterprise standards for calendars.

Stay tuned for more updates.

Sample from Gamma distribution in Java

2007-11-28T12:49:00.000Z

Back to the Java struggles.
Today I am giving away a piece of Java code which allows one to sample a random variable from Gamma distribution.

import java.util.Random;

public class Samplers {
  private static Random rng = new Random(
      Calendar.getInstance().getTimeInMillis() +
      Thread.currentThread().getId());

  public static double sampleGamma(double k, double theta) {
    boolean accept = false;
    if (k < 1) {
 // Weibull algorithm
 double c = (1 / k);
 double d = ((1 - k) * Math.pow(k, (k / (1 - k))));
 double u, v, z, e, x;
 do {
  u = rng.nextDouble();
  v = rng.nextDouble();
  z = -Math.log(u);
  e = -Math.log(v);
  x = Math.pow(z, c);
  if ((z + e) >= (d + x)) {
   accept = true;
  }
 } while (!accept);
 return (x * theta);
    } else {
 // Cheng's algorithm
 double b = (k - Math.log(4));
 double c = (k + Math.sqrt(2 * k - 1));
 double lam = Math.sqrt(2 * k - 1);
 double cheng = (1 + Math.log(4.5));
 double u, v, x, y, z, r;
 do {
  u = rng.nextDouble();
  v = rng.nextDouble();
  y = ((1 / lam) * Math.log(v / (1 - v)));
  x = (k * Math.exp(y));
  z = (u * v * v);
  r = (b + (c * y) - x);
  if ((r >= ((4.5 * z) - cheng)) ||
                    (r >= Math.log(z))) {
   accept = true;
  }
 } while (!accept);
 return (x * theta);
    }
  }
}

The last hope for working with Exchange

2007-11-21T00:54:00.000Z

Following on the topics of using iCal with the Exchange server. AppleInsider published a very interesting comparison of iCal and Mail.app to the coming Entourage 2008.

The AppleInsider Article left me very upset, as it is evident that there will be no improvement to synchronisation with iCal. Hopefully Office will become much faster, as it will be compiled as a universal binary...

The only remaining hope is for Snerdware's GroupCal. Guys from AppleInsider got it wrong, as GroupCal is not yet working with Leopard. We have to wait a little, and hope that the developers from Snerdware solve their problems and release a new version soon.

Forget my complaints about the price. I will buy the new GroupCal on the first day it is released.

Boeing CalDAV proxy for Microsoft Exchange

2007-11-08T10:57:00.000Z

I concluded the previous post with a rumor that Boeing have developed a CalDAV proxy for Microsoft Exchange. This was my last hope for using iCal with my work Exchange account.

Unfortunately, this is the dead end. I have found what this software is and why it will never give me the functionality I want.

First of all, Boeing have developed a Free-Busy connector for Exchange, not a complete CalDAV proxy. This is a web application (an ASP by the way) which takes someones e-mail as an input and returns their Free-Busy information in a format compatible with some CalDAV servers. I have tried using it with Bedework CalDAV server and I found it absolutely useless. This is a very good first attempt, but there are the following shortfalls:

You need to run this special connector service, and you need to run it on a Windows machine.
This initial implementation does not have any built-in security. It just accesses the Exchange server through one preconfigured account. You canot use different users for different requests
This will only allow you to see whether john@example.com is free or busy from 9am to 10am, but would not let John know if you are busy.
This example service in not available in the wild.

On the other hand, this is a good first step towards integration with Exchange. If the community demonstrates any interest, there will be someone to make further road.

How to synchronize iCal with Exchange?

2007-10-30T20:13:00.001Z

Apple iCal is a very neat calendaring application. I love it. It synchronizes with my iPhone. It works reliably. The new version supplied with Leopard has everything I need: scheduling appointments, booking rooms, inviting other people, tells other people when I am busy or free.

Though, it is not compatible with Microsoft Exchange server our University uses.

I have run a little investigation on how to resolve the problem, and I'd like to share my finding with you.

Solution 1: Use Microsoft Entourage.

Entourage is a part of Microsoft Office for Mac. It is a Mac version of Outlook. It will allow you to manage your mail and calendars well. You can even synchronise your calendar with iCal! Automatically!
Just go to Entourage>preferences, and select Sync Services.
Now, on the right, you can ask Entourage to synchronise your calendar with iCal. This will create an additional calendar in your iCal called Entourage. It gives you three options.

Take verything from Exchange and put it into iCal,
take everything from that special calendar in iCal and put it into Exchange,
Merge the calendars, and make sync work both ways.

If you want to book a room, you can do it with Entourage. If you just want to notify your colleagues that you are not available for a meeting every workday from 1pm to 2pm, just create a recurring appointment called "Lunch" and they will know you are busy.

If you are happy with this solution, go for it! It will even synchronise with your iPhone! Thanks for coming, and you know how to make a blogger happy - read carefully what Google guys suggest to you in the right part of the page.

Now, why I don't like Entourage.

First of all, I just love iCal and Mail.app. I do not want to use Entourage, even though I get it for free through the site license.

Second, the current version of Entourage was written years ago, when Macs were made with PowerPC processors. Those processors spoke a different language to what Intel processors speak. If you run Entourage (which is written in the ancient language of PowerPC) on you shiny new Intel Mac, you use a translator with a Rosetta Stone (go and find what it is in Wikipedia, it is interesting). The point is that the translator is slow. And iCal is written in plain English, oops... in plain Intel, and it works natively.

Third, Entourage can only synchronize your calendars while it is running. You cannot say it synchronize, and go on using iCal and mail.app alone. You still need to run Entourage. Though, you can run it hidden, press Command-H when Entourage is open and it is now hidden.

Fourth.... I do not like Entourage. I do not want to support Microsoft... I want everything to be simple....

We all hope that Microsoft will update their Office for Mac early next year, and hopefully some of my problems will be resolved. I am not sure, though, if it is worth the money people pay for it.

Oh, if you are a representative of Microsoft marketing, please, contact me and I will let you know what I want for Christmas.

If you are still with me, than you are probably looking for an alternative, so there you go:

Solution 2: Use GroupCal from Snerdware

The guys have made quite a job. It really works! Everything I ever wanted. Though, it does not work under Leopard.... yet...
Go and see for yourself: www.snerdware.com/groupcal

I would be happy, but...

It does not work with Leopard.... yet...

The way you see other people's free/busy time, and if the room is available is not nice. How I wish it worked just exactly like iCal in Leopard promises...

I do not have a site license, and my Boss will tell me to go and use Windows. So, paying $55 from my own pocket is unpleasant. Please, fix the problems and make it a 20...

So, I went on went on with my investigation

Solution 3: Use open standards! Like iCal Server!

Oh, if you are powerful enough, you can save yourself quite a buck by not using Exchange at all! Go for CalDAV solution which will do everything you ever waned for free. It will work seamlessly with iCal in the end. And remember, iCal Server is open source, you can install it on one of your Macs for free. Whatever Apple charge for OS X Server, they charge it for the convenience of seamless installation. Contact the (smart) tech support.

This was not an option for me... so I went on and discovered another twisted solution.

Solution 4: iCal to Google Calendar to Outlook to Exchange. All aboard!

Bad news - you won't be able to see busy/free time of other people, and you still have to pay $50.
Good news - it will also synchronise with Google Calendar, and even send you SMS with notifications if you want.

The solution consists of two parts. The first one is to use magical Spanning Sync. This is a little program which will synhronize your iCal with Google Calendar. Works like a charm. Go for it even if you do not want anything else. Leopard compatible. Best program you can buy for your Mac, and just $25 for a year.

The second part is to use SyncMyCal tool from www.syncmycal.com on a Windows machine. I have enough of old computers I can hide in a cupboard at work which will run a scheduled synchronisation of Google Calendars with my Exchange...

Well... maybe Entourage is not all that bad after all... This is a really twisted solution. I am not going to use it.

Solution 5: Use Outlook Web Access

This is a web application to access your mail and calendars on the Exchange servers. Many companies have it. Clumsy and awful... I will use it only if I have no other choise whatsoever.

Solution 6: Do it the geeky way
Here is how: http://www.macosxhints.com/article.php?story=20041110141248547

I have to admit that I never tried this idea. Tell me if it works for you.

Solution 7: Hosted services

There are a number of synchronisation services, free and commercial, which will allow you to synchronise your calendars across different platforms. I have tried these:

Xchange network
ScheduleWorld

The main problem with these is that I do not want my calendars to be hosted somewhere else. I do not want even a potential breach of my privacy.

Now iCal supports proper communication with Microsoft Exchange server. If you are looking for hosted Exchange solution check out SherWeb at http://www.sherweb.com/

Nonexisting Solution: CalDAV connector to Exchange

This is what I would like to see. I want to have a CalDAV server, like iCal Server from Apple, able to fetch all the data from Exchange. This has to work transparently, as if I am using CalDAV.

The project itself does not sound impossibe. There are open source CalDAV implementations. There is Evolution Exchange Connector, an open source client for Microsoft Exchange. The final step is to make a transparent implementation of CalDAV server which talk to Exchange. Everyone will be happy then.

If you are a developer, and you can make this happen, then please, do! Make it cheap, like $20-$30, and you will sell a ton of copies. Send me a cheque for the idea if it works out well. Seriously, contact me if you are interested in this development, I can contribute to requirements. We can make millions happy.

Conclusions

I ended up printing out my calendar and sticking it to my door. People have to ask me if I am free at any particular moment, and I will check with my iCal. I do not synchronise with Exchange.

I really hope, the whole problem can be resolved in the year to come.

So far... dead end....

Thanks for reading. I guess, there must be some other solutions. Please, share your experiences and ideas in the comments. Check out what Google suggests in the right pane, they may have found a better solution elsewhere.

I do not hold any stock in any companies mentioned in this article. Your feedback is always welcome.

Cheers!

PS: I just have found that Boeing claim they have implemented an Exchange connector for CalDAV. I could not find any more details about this rumor.

Curvy tabs in your Eclipse RCP Application

2007-10-09T13:49:00.000+01:00

You might have noticed that the default look and feel of your Eclipse RCP application is slightly different to what you are used to while using Eclipse IDE. Many developers want to have exactly the same "Curvy" shape of tabs for the views instead of "lame" square ones. There are two ways of achieving that.

Way 1: The proper one. Depending on the version of Eclipse platform you are using, you have to edit (or create) either preferences.ini or plugin_customization.ini file in the root of your defining plugin. Add the following line to that file:
org.eclipse.ui/SHOW_TRADITIONAL_STYLE_TABS=false
and voila!

Way 2: The working one. I have no idea why the previous (proper) way of parameter definition fails sometimes. I have found a neat fix for the problem: you might want to set this workbench preference in your application workbench window adviser.
In ApplicationWorkbenchWindowAdviser.java add the following line:

PlatformUI.getPreferenceStore().setValue(IWorkbenchPreferenceConstants.SHOW_TRADITIONAL_STYLE_TABS, false);

to public void preWindowOpen() method. and everything will begin to work like magic!

Good luck!

Dynamic Palettes in GEF

2007-10-04T16:38:00.000+01:00

Hello again, I am finally back from holiday. Today I will describe a short trick for dynamically updating your GEF Palette.
I wanted to change the contents of the palette in graphical editor depending on user's choice in one of the combo boxes. Imagine combining two models into one by giving user a choice which one to use.

The original palette is usually created on graphical editor initialisation. You, however, can change the contents of this palette dynamically by modifying the PaletteRoot of your graphical editor.

So, I have created an additional method in my graphical editor:

public void updatePalette(LogicalModel model) {
  PaletteRoot root = getPaletteRoot();
  List children = root.getChildren();
  for (int i = 0; i < children.size(); i++) {
   if (children.get(i) instanceof PaletteGroup) {
    PaletteGroup entry = (PaletteGroup) children.get(i);
    if (entry.getLabel().equals("Plots")) {
     // This is a group we want to update
     List plots = model.getPlots();
     int size = entry.getChildren().size();
     for (int j = 0; j < size; j++) {
      entry.getChildren().remove(0);
     }
     for (int j = 0; j < plots.size(); j++) {
      CombinedTemplateCreationEntry component = new CombinedTemplateCreationEntry(plots.get(j).getName(), plots.get(j).getName(),
        Plot.class, new SimpleFactory(Plot.class), ImageDescriptor.createFromFile(ResultsEditor.class,
          "icons/plot.gif"), ImageDescriptor.createFromFile(ResultsEditor.class, "icons/plot.gif"));
      entry.add(component);
     }
    }
   }
  }
 }

This method flushes the whole content of the palette group named "Plots", and recreates it from some logical model.
All I have to do now is to call this new method from an appropriate listener to combo's selection change.

Storing binary information in XML

2007-08-15T10:14:00.000+01:00

Sometimes we need to store some binary information within XML format, which is textual.
Designing schemata for your data you should avoid using binary representations if possible, but there are some cases when you absolutely need it. For example, what if you want to store a picture within your XML?

The problem is that binary data may contain special characters which are prohibited for use in XML.

The solution to this problem is to recode your binary data into a textual form using, for example, Base64 or UUEncode algorithms. The following fragment of Java code demonstrates how to convert an array of bytes (a chunk of binary data) into a properly formed String using Base64 algorithm.


byte[] data = getMyBinaryData();
String encodedData = new sun.misc.BASE64Encoder().encode(data);

You then can use encodedData as a value for some XML element.

The reverse process of decoding a String into an array of bytes can be done using the following Java code:


String encodedData = getEncodedData();
byte[] data = new sun.misc.BASE64Decoder().decodeBuffer(encodedData);

UUEncode/UUDecode algorithms are also available in sun.misc package.
This approach will, unfortunately, fail in an Applet, as the access to sun. packages is prohibited for applets. You have to use a different implementation of a transcoding algorithm in such cases. One of the alternatives is: http://iharder.sourceforge.net/current/java/xmlizable/

Computing MD5 digest (checksum) in Java

2007-08-01T12:03:00.000+01:00

MD5 digests are useful to track file (or, in fact, data) modifications. It is very unlikely that two files will have the same MD5 digests. It is extremely unlikely that a minor modification to a file will preserve its MD5 digest.

I use MD5 digests to decide whether a precomputed result is still valid. For example, you simulate a system of differential equations defined in file "system.xml", and you save the result to a file "result.xml". But what if the user modifies the original system of equations? The computed result will not be correct anymore. The solution is to store the MD5 digest of the original system of equations with your result. When result is selected for display, you check whether the digest of your current system of equations matches the one stored with the result, and if they don't match, the result is not valid.

There is a number of ways to compute MD5 digests. You can find hundreds of implementations on the web. I stick to the one which comes with standard Java Runtime Environment distribution from Sun Microsystems. Below is my function which returns a String representation of MD5 digest for an arbitrary file:


public String checksum(File file) {
  try {
    InputStream fin = new FileInputStream(file);
    java.security.MessageDigest md5er =
        MessageDigest.getInstance("MD5");
    byte[] buffer = new byte[1024];
    int read;
    do {
      read = fin.read(buffer);
      if (read > 0)
        md5er.update(buffer, 0, read);
    } while (read != -1);
    fin.close();
    byte[] digest = md5er.digest();
    if (digest == null)
      return null;
    String strDigest = "0x";
    for (int i = 0; i < digest.length; i++) {
      strDigest += Integer.toString((digest[i] & 0xff) 
                + 0x100, 16).substring(1).toUpperCase();
    }
    return strDigest;
  } catch (Exception e) {
    return null;
  }
}

And if you are using Eclipse RCP and want to compute MD5 digest for an IFile object, here you are:


public String checksum(IFile file) {
  try {
    InputStream fin = file.getContents(true);
    java.security.MessageDigest md5er =
        MessageDigest.getInstance("MD5");
    byte[] buffer = new byte[1024];
    int read;
    do {
      read = fin.read(buffer);
      if (read > 0)
        md5er.update(buffer, 0, read);
    } while (read != -1);
    fin.close();
    byte[] digest = md5er.digest();
    if (digest == null)
      return null;
    String strDigest = "0x";
    for (int i = 0; i < digest.length; i++) {
      strDigest += Integer.toString((digest[i] & 0xff) 
                + 0x100, 16).substring(1).toUpperCase();
    }
    return strDigest;
  } catch (Exception e) {
    return null;
  }
}

Eclipse JRE Configuration to work with Mac OS X

2007-07-30T23:55:00.000+01:00

Just a short tip for today. When you are configuring your Eclipse framework (Preferences/Java/Compiler) or a particular Java project, select the compiler compliance level to 5.0, as JDK 6.0 is not yet available on Mac OS X (it is still in a beta state, so everything may change soon). This ensures that your final product will run on Mac OS X when you export it in the end.

JNI on Mac OS X

2007-07-27T14:56:00.000+01:00

JNI (Java Native Interface) is a way to use native (read "C") functions in your Java program. You can read more details about this approach at Sun Java Web Site.

A particular problem I was struggling with is how the libraries are loaded when running on Mac OS X. The standard naming convention for using the dynamic libraries in Mac OS X is to use "lib" prefix, and ".dylib" as the suffix. Java Virtual Machine, however, does not follow this convention when trying to load a library, and requires you to use ".jnilib" suffix (and "lib" prefix) instead.

The typical loadLibrary call:
System.loadLibrary("mine");
will search for libmine.jnilib in your java.library.path; and if you have named your library libmine.dylib, it will produce you an UnsatisfiedLinkError exception.

Eclipse SWT: Disabling selection event in a Table

2007-07-19T16:24:00.000+01:00

Sometimes we do not want users to be able to select cells or rows in the tables, but the standard styles for SWT Table widget do not give a NOSELECTION option.

This, however, can be easily achieved by masking the selection event, Just use the following piece of code for creating your tables:

final Table table = new Table(group, SWT.SINGLE | SWT.BORDER);
table.addListener(SWT.EraseItem, new Listener() {
    public void handleEvent(Event event) {
        if((event.detail & SWT.SELECTED) != 0 ){
            event.detail &= ~SWT.SELECTED;
        }
    }
});

Eclipse RCP: NullPointerException when creating the SaveAs dialog

2007-07-17T14:58:00.000+01:00

When using standard editors from org.eclipse.ui.ide, the default Save As action usually fails, throwing a NullPointerException. This happens when the program tries to create a Save As Dialog. A particular exception usually looks something like:

java.lang.NullPointerException
    at org.eclipse.ui.dialogs.SaveAsDialog.createContents(SaveAsDialog.java:102)
    at org.eclipse.jface.window.Window.create(Window.java:426)
    at org.eclipse.jface.dialogs.Dialog.create(Dialog.java:1124)
    at org.eclipse.ui.texteditor.AbstractDecoratedTextEditor.performSaveAs(AbstractDecoratedTextEditor.java:1580)
    at org.eclipse.ui.editors.text.TextEditor.performSaveAs(TextEditor.java:115)
    at org.eclipse.ui.texteditor.AbstractTextEditor.doSaveAs(AbstractTextEditor.java:3538)
    at ....MultiPageEditor.doSaveAs(MultiPageEditor.java:196)
    ....

This, actually, happens when the dialog tries to set the banner image for the dialog, but the image is not available. The default IDEWorkbenchAdvisor registers such an image for the default IDE, so must your application. The solution is the following:
1) Make sure you add org.eclipse.ui.ide to the dependencies of your plugin
2) In your WorkbenchAdvisor class add the following code to the initialise() method implementation:

Bundle ideBundle = Platform.getBundle(IDEWorkbenchPlugin.IDE_WORKBENCH);
    final String banner = "/icons/full/wizban/saveas_wiz.png";
    URL url = ideBundle.getEntry(banner);
    ImageDescriptor desc = ImageDescriptor.createFromURL(url);
    configurer.declareImage(IDEInternalWorkbenchImages.IMG_DLGBAN_SAVEAS_DLG, desc, true);

This will declare a standard banner image from the org.eclipse.ui.ide plugin as the proper banner image for SaveAs dialog in the same way as it is done in the IDEWorkbenchAdvisor. Running your application again demonstrates the correct behaviour. If you are still getting a NullPointerException, check whether you declared plugin dependencies, and whether you included org.eclipse.ui.ide to your target platform.

A follow-up to SBML-related post

2007-07-11T13:03:00.000+01:00

I contacted the SBML team to discuss the problems I wrote about last week. And I really appreciate their prompt response and collaboration. Especially, I want to thank Michael Hucka, who clarified a lot of things to me.

First of all, as Michael pointed out, XML Schema standard requires the targetNamespace to be an URI, which does not mean this should resolve to a real web resource. So my reasoning about what schemata you obtain when trying to fetch the namespace URI does not hold water.

More importantly, is what are the target namespaces for different SBML standards:
Level 2 Version 1 http://www.sbml.org/sbml/level2
Level 2 Version 2 http://www.sbml.org/sbml/level2/version2
Level 2 Version 3 http://www.sbml.org/sbml/level2/version3

So, libSBML version 2.3.4, indeed, supports Level 2 Version 1 specification only, and it crashes when other version is processed. The SBML team is working on fixes to this problem, and a pre-release version of libSBML 3.0.0 works with the newer specification perfectly fine. You code, however, might need some changes due to architecture redesign in libSBML.

Michael also agreed that it won't hurt to make the web server serve different versions of the SBML schema to different URLs, so the schemata are now served as I (and JAXB) expected.

Summary:

Check the schema twice, and make sure that you are really using the proper schema.
If you need to support SBML L2 V 2 or 3 - use libSBML 3.0.0
Thanks to everyone from the SBML team for your help

SBML standard and problems with libsbml

2007-07-07T14:50:00.001+01:00

I have spent several days fighting a problem with my Java program which uses libsbml indirectly through the Java Native Interface. Every once in a while the whole java virtual machine would crash while reading an SBML model. So, I performed a detailed investigation of what is wrong about libsbml and the standard itself.

Let's begin with an SBML schema. My program was using SBML Level 2 Version 2 schema, and the most recent one is SBML Level 2 Version 3. I specifically checked if this problem persists, and my comments are still valid. So, the key point. The schema defines a specific XML namespace for SBML, and for SBML Level 2 Version 3 it is:

targetNamespace="http://www.sbml.org/sbml/level2/version3"

However, libsbml, being a de facto standard library for SBML support crashes badly when processing a model compliant to the standard. I begin my model with the following:

<?xml version="1.0" encoding="UTF-8"?> <sbml xmlns="http://www.sbml.org/sbml/level2/version3" level="2" version="3">

Which is exactly what the schema prescribes. Next, I am trying to read this model with readSBML example program supplied with libsbml. Guess what happens?

$ ./readSBML model1.xml Segmentation fault

Too bad, as this means that the problem is not with my code, but with libsbml itself. Running readSBML with valgrind demonstrates obvious memory management errors in libsbml's XML parser. The problem persists with both expat and xerces-c implementations.

But why the majority of software tools work fine with this implementation of XML parser? I went on and took a look on what Copasi produces when exporting a model to SBML. Surprisingly, the model produced with Copasi does not comply to the SBML standard. And the root element of the produced model uses the wrong namespace:

http://www.sbml.org/sbml/level2

I tried to feed that (non-compliant) model to readSBML, and everything went fine. I even tried to valgrind readSBML with the new model and look:

==15583== ERROR SUMMARY: 0 errors from 0 contexts ==15583== All heap blocks were freed -- no leaks are possible.

Unbelievable! But is it a problem with Version 3 (and Version 2) schema support or with the whole SBML standard? Just type that namespace http://www.sbml.org/sbml/level2 in your browser, and it will show you the proper schema. The schema, however, clearly specifies:

targetNamespace="http://www.sbml.org/sbml/level2/version3"

So, the standard itself is inconsistent, and standard support is badly broken.

What can we do at the moment? Well, the only way around is to break the standard and continue using the wrong namespace. Does it require any attention from the SBML consortium? I think so. What can I do to write an SBML compliant software tool? Write my own SBML support library. Will a properly SBML compliant tool read the vast majority of published models? I doubt it.

What have I done to fix my problem? I produced a little routine which converts proper models to the broken form and then reads them with libsbml parser.

Valgrind it

2007-07-06T10:22:00.001+01:00

A student was executing his tasks on the same machines as mine, and over the night his task allocated almost all the available memory causing incredible amount of swapping and stopping any progress of all the tasks (not to mention that he started the task by directly logging in to a computer rather than through a queue). Here is a very important advice, and if you follow it you can avoid upsetting many people in a shared computing resources environment: always check your program for memory management errors and memory leaks. It is pretty easy to do with a modern Linux system, just use a tool called valgrind.

I recommend to start in a standard way, and read valgrind manual page, just type
$ man valgrind
in your terminal.

Going beyond referring users to a manual page, here is a little example of how this tool can be used. Imagine starting with the following C program:

#include <stdio.h> #include <string.h> #include <stdlib.h> int main(int argc, char** argv) { char* string = (char*) malloc(sizeof(char)*strlen(argv[0])); strcpy(string,argv[0]); printf("My name is %s\n",string); return 0; }

Well done! You have won 5 points if you noticed all the three problems. Let's imagine you haven't.
First, you compile your program with debugging options:
$ gcc -ggdb ex1.c -o ex1
No compilation time errors, no warnings. If you run the program, the result usually depends on how lucky you are. I am lucky, and the result is exactly what I expected:

$ ./ex1
My name is ./ex1

But something can be wrong about this program. Let's use valgrind to check. Start with the default checking, and run:
$ valgrind ./ex1
The tool detects 2 errors and a memory leak of 3 bytes. The first error is:

==10523== Invalid write of size 1 ==10523== at 0x4006A2C: strcpy (mc_replace_strmem.c:272) ==10523== by 0x8048406: main (ex1.c:7) ==10523== Address 0x401F02B is 0 bytes after a block of size 3 alloc'd ==10523== at 0x4005400: malloc (vg_replace_malloc.c:149) ==10523== by 0x80483EF: main (ex1.c:6)

So, at line 7 of my program I am writing 1 byte beyond the allocated memory block. Did you read Kernighan-Ritchie book? Every string has to be ended by a zero (\0) character. This character is not counted when computing the length of a string, but is copied when copying a string.
The second error is:

==10523== Invalid read of size 1 ==10523== at 0x4006283: strlen (mc_replace_strmem.c:246) ==10523== by 0xB2A0C1: vfprintf (in /lib/libc-2.5.so) ==10523== by 0xB2F602: printf (in /lib/libc-2.5.so) ==10523== by 0x8048419: main (ex1.c:8) ==10523== Address 0x401F02B is 0 bytes after a block of size 3 alloc'd ==10523== at 0x4005400: malloc (vg_replace_malloc.c:149) ==10523== by 0x80483EF: main (ex1.c:6)

this is directly caused by the previous one. That 1 byte written outside of the memory block is now read when printing the string out. So, I should fix my program by extending the size of the allocated memory block. I create a new program called ex2.c:

#include <stdio.h> #include <string.h> #include <stdlib.h> int main(int argc, char** argv) { char* string = (char*) malloc(sizeof(char)*(strlen(argv[0])+1)); strcpy(string,argv[0]); printf("My name is %s\n",string); return 0; }

See? I added 1 symbol to the end of my string. Compile and run the program in exactly the same way.
Running with valgrind produces the following output:

==10812== ERROR SUMMARY: 0 errors from 0 contexts

Congratulations, we fixed both of the errors. But there is a little more to this program:

==10812== LEAK SUMMARY: ==10812== definitely lost: 6 bytes in 1 blocks. ==10812== possibly lost: 0 bytes in 0 blocks. ==10812== still reachable: 0 bytes in 0 blocks. ==10812== suppressed: 0 bytes in 0 blocks. ==10812== Use --leak-check=full to see details of leaked memory.

Let's follow the advice, and run leak detection tool:
$ valgrind --leak-check=full ./ex2
It produces the following report:

==10932== 6 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==10932== at 0x4005400: malloc (vg_replace_malloc.c:149) ==10932== by 0x80483F2: main (ex2.c:6)

This means that 6 bytes allocated with our malloc in line 6 were never released. Adding
free(string);
to the end of the program (and naming it ex3.c) gives us an ideal result:

==11013== ERROR SUMMARY: 0 errors from 0 contexts ==11013== All heap blocks were freed -- no leaks are possible.

Now you can be more sure that your program is correct.

A couple of closing remarks.

Remember, valgrind checks only that part of your program which was executed. If you have several branches (if-then-else) or subroutines, make sure that you have a decent set of test scenarios to cover it all. You might want to search Google for test coverage methodologies (just type man gcov if you do not know how to use Google)
Leaving such dynamic memory management errors causes segmentation faults. The most nasty ones appear when such a problem is located inside a shared library called from Java native method. The java virtual machine will crash leaving you frustrated and incapable of catching an exception or something to recover your program on the fly.
Just having a memory leak in a shared library will eventually waste all your virtual memory, and the system will need to be rebooted. Guess why Windows servers have to be rebooted every month or so?

Starting the Blog

2007-07-05T19:40:00.000+01:00

This time I am starting a real blog, devoted solely to the professional questions and tricky problems I have to deal with every day. Hopefully my enthusiasm will last a little longer this time.

Stochastic Processes and Distributed Computations

2007-07-05T11:43:00.000+01:00

The most disturbing problem I am struggling with way too often is related to the way random numbers are generated in the majority of modern computers. As many of you know, those are not really random, and are generated with some kind of a numerical algorithm. Most of the random number generation algorithms rely on something called "random number generator seed", that is a parameter to such an algorithm to start a particular pseudo-random sequence. But beware - the majority of algorithms will spit out exactly the same random sequence when given the same seed. The traditional solution for this problem is to initialise the seed with the current time (when the program is started), something akin to:


gsl_rng_set(r,time(NULL));

if you are using C and GSL.

This, however, will not help if you are running distributed computations on a cluster, and you need to get an independent random sequence for each of the processes. The solution in this case is to generate a unique random seed for each of the processes. Adding the rank of the current job to the current time value works the best for me when using MPI. The solution looks like the following:
gsl_rng_set(r,time(NULL) + MPI::COMM_WORLD.Get_rank());
for C++, and

int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
gsl_rng_set(r,time(NULL) + rank);

if you are using C.

If you are not using MPI, but rather run independent jobs through the execution queue, you cannot compute such a rank, but you can use hostname checksum / process id combination instead. It is extremely unlikely (I am speaking about the most UNIX architectures) that two instances of your program will be run on the same host within the same second and have exactly the same process id. If it is the case, you have to think a little, but you already have the general idea.