Managing Really Large Data Sets

For a while now I’ve been working with next generation sequencing datasets.  Each dataset consists of around 10 million mapped genome positions, and an experiment can consist of 10 or more datasets. When analysing this data memory usage is a major issue.  Up until now our approach has been to try to store everything in…

Date
Categories
Tags
Comments

Published:August 27, 2009 View Post

Bioinformatics

Comments closed

Optimising Java Memory Usage

In the application I’m working on I have to deal with large amounts of data, which means handling tens of millions of java objects.  One of the biggest problems we face is keeping the memory usage of the program under control. The main object which accounts for the vast majority of the memory consumption contains…

Date
Categories
Tags
Comments

Published:June 21, 2009 View Post

Computing

Comments closed

Exporting SVG from Java

Several programs I’ve written in Java have had an image export componet to them.  Up until now the export has always only been as a bitmap image.  This is very easy to do using a BufferedImage and an ImageWriter.  Given a Component (all AWT or Swing widgets) and a file you can create a PNG…

Date
Categories
Tags
Comments

Published:February 8, 2009 View Post

Computing

Comments closed

Broadcom problems under linux

I’ve been trying to install Fedora 9 on a Dell PowerEdge T100 and have been tearing my hair out trying to get the network card to work. The card in question is a Broadcom Netextreme BCM5722 Gigabit Ethernet PCI Express adapter.  I’ve used similar Broadcom cards in other machines without any problems, but this time…

Date
Categories
Tags
Comments

Published:November 20, 2008 View Post

Computing

Comments closed

Troubleshooting IMAP SSL

I’ve just spent a while trying to troubleshoot my SSL IMAP connection.  This is the first time I’ve had to do any diagnostics since switching to an SSL secured mail connection. When my connection stopped working I got only a very non-specific error from OSX mail, and no error at all from Thunderbird (it just…

Date
Categories
Tags
Comments

Published:November 5, 2008 View Post

Computing

Comments closed

Fighting Apple over SSL mail

As I’ve been setting up this domain I’ve been trying to get both my MacBook and my iPod touch to talk to my new mail servers.  You’d have thought that this should have been pretty straightforward (I did). Since I have the option of using either unencrypted or SSL protected connections to the mail servers…

Date
Categories
Tags
Comments

Published:October 10, 2008 View Post

Computing

Comments closed

Swing Speedup!

I’d never really been one to subscribe to the popular perception that swing is so horrifically slow that it isn’t suitable for use in any serious application.  Most of the problems I’ve seen with swing based programs have resulted from people not knowing how to make processing tasks run in a separate thread and therefore…

Date
Categories
Tags
Comments

Published:October 10, 2008 View Post

Computing

Comments closed