Migration is one of the more continual banes of my life and its one of the more interesting problems. Migration from Joomla! 1.0 to 1.5 combined many many changes. Not only did we alter the way the system worked in places in a somewhat drastic way incidentally removing features but we forced everyone for the first time to consider their encoding properly. A lot of sites were using the default latin1 encoding, even if they weren’t actually a Latin 1 language. Some sites used Latin 1 but some how in their database was actually UTF-8 characters every so often.

When I sit here and consider this, I realise that we really did make the right decision to almost force people to create new databases and new tables that were in UTF-8. It means that all of this ‘fun’ with encodings won’t happen unless we move from UTF-8 to UTF-16 or something similar in the future. Whilst its an incredible amount of pain right now, once we’re over this hump it is for the most part plain sailing.

The other interesting aspect of the process is the design of the migrator itself. When David Gal released the first migrator, it was basically a SQL dumping tool that was specifically designed to work with only the core tables that shipped with Joomla1 and to move those aross into the new site via an SQL dump. The design of the migrator that I built took a similar approach of making an SQL file that could be used to install items but it also addressed the concern that third party developers couldn’t extend the original version. My version of the migrator was heavily based around the concept of an ‘ETLPlugin’. An ETL plugin was a simple PHP class that at the bare minimum provided a descriptive display name, the name of the table in question and that was it. Presuming you had no changes in your database, writing a plugin was quite simply four lines of code:

class TableName_ETL extends ETLPlugin {
      function getName() { return "TableName ETL Plugin"; }
      function getAssociatedTable() { return 'tablename'; }
}

Thats it, thats an ETL plugin. The migrator has code that will find the columns in the table and build the appropriate SQL statement to get it into the system. You can add stuff like “getSQLPrologue” to prepend stuff before your rows such a “CREATE TABLE” statement.

As far as I can tell, the only vendor who actually made an ETL Plugin was DocMan. After complaining that they didn’t have a method of extending the tool to support themselves, they didn’t even bother to use it. Documentation on ETL plugins is in my Migrator RC1 OpenOffice.org (ODF ODP) presentation that was dated August 2007, that is around 5 months before we shipped 1.5 – and even now one year later I am yet to see any more plugins. Doing a Google search on “joomla 1.5 etl plugin” reveals links to the official documentation and copies there of, one link to the ODP file mentioned above with “Developing ETL plugins” in Google’s abstract and forum posts for Azrul, CommunityBuilder and a Real Estate component all mention “manually” migrating the users data from the database stating they don’t need an ETL plugin. For some reason they wish to make their users lives harder by forcing them to go outside of the system that I put together to make their lives easier. They could write an ETL plugin really easily and not have to worry about doing anything but shifting files manually (something mostly easily achieved with most file browser/FTP tools).

At the end of the day its sad to spend time building a system that will handle their needs almost automatically but still deliver flexibility if they wanted it and then have it not used and have them say to use a manual migration anyway. Perhaps I’ll write a post on how to make an ETL plugin (again).

For the last few days I’ve been wondering about getting XDebug up and running on my Mac with Leopard since it doesn’t ship natively. Now a few of the guides I saw suggested that I use the Komodo one that was already precompiled for Mac OS X, great! I wasn’t particularly in the mood to source compile something so I went off and downloaded it and then dumped it into the default PHP extension directory for Leopard (/usr/lib/php/extensions/no-debug-non-zts-20060613/), reconfigured php.ini to add an xdebug section and load the file and restarted Apache. No joy, the module wasn’t in Joomla!’s PHP Info page when it should be. I fired up Console and looked for Apache’s log files and found this charming looking message:

Failed loading /usr/lib/php/extensions/no-debug-non-zts-20060613/xdebug.so:  (null)

A quick Google resulted in a few blog posts and similar that basically had the same conclusion: compile from source. One blog post recommended to source compile but noted that the command line version didn’t work when the web server worked and another one pointed out that Komodo’s version was i386. This got me thinking and I did a quick check. Under Leopard, Apache (httpd) is a 64 bit application (check it out using Activity Monitor, enable the “Kind” column and it’ll say “Intel (64-bit)”) however if one runs “file” over the default PHP they will find its a universal binary compiled for PPC and i386, no x86_64.

So the solution to the dilemma of it not working properly is to build a universal binary with all of the platforms we care about, which is what I’ve done. I cross built the module for PPC, i386 and x86_64 individually and then used “lipo” to create a fat binary called “xdebug.dylib”. I then took this fat binary, dumped it into the extensions directory and loaded it using the “zend_extension” command. I restarted Apache and checked the logs for the error and found none. I then looked at PHP Info in Joomla! again and this time I xdebug was there. On the command line I ran “php -m” and noted that xdebug was now in the “Zend Modules” list. Unfortunately my old PPC G4 died (RIP) so I can’t test this on a PPC Leopard machine (my work one is still Tiger unfortunately). It should work mind you even though I’ve not tested it.

You can download the file with some install instructions from JoomlaCode: http://joomlacode.org/gf/download/frsrelease/9372/35384/xdebug-2_0_4.zip

In other news I spent some more time today looking over 1.6 and thinking about how I’m going to handle the file backup aspect of the system. I’m considering using a tarball style system to create the files and working through stuff like what options are required (exclusion filters? inclusion filters? exclude paths or include paths etc). I’ve written down some thoughts and I’ll probably continue working on it on Monday. For the weekend I have on my list getting some work done on my LDAP User plugin and releasing JAuthTools’ 1.5.4 release. I’ve got a test site arranged for myself so I’ll be able to play around with things which should be good. An exciting weekend ahead (at least for me anyway!).

Today I spent some time looking over the 1.6 backups code where I’d let it off a while ago and managed to get it working. There were a few really simple things wrong from when I’ve changed something in one place but not another but now its up to the point that its working properly so I’m happy. Annoyingly I keep using an old 1.5 instance to restore from so I’m going to have to do a new 1.6 instance and use that for my new dump files. Mind you I’m happy that this little side project has come together, once I get the file system level backups working properly (at the moment it does in-database table copies and SQL dumps/loads) I’ll be able to put it together into the installer subsystem and have full packages made of items before install. I’m still trying to work out what Joomla! upgrades look like yet, but I’ll figure it out sooner or later.

Almost depressingly the most interesting thing of today was the meeting on what services we were going going to deliver to the regions. We are moving in to try and unify the applications and deliver them at a reasonable pace to the users. The main problem is that people have applications out there already but their data is here and in the short term we’re not going to be able to deliver the links possible for them to reasonably maintain those services locally anyway…kind of sucks. For the majority of users we can deliver applications via Citrix in the short to medium term (we’re already doing this and its working for the most part) however there are some applications, particularly the Engineering ones (GIS mapping tools and AutoCad). These are applications that rely on dedicated graphics hardware and don’t fit in well with the Citrix style environment (or any form of remote environment for that matter). So we’re looking at different options such as VDI or VNC to handle those particular people. Where possible the main aim is not to move infrastructre out into the regions and keep it centralised. Wonder how long we’ll stick to that.

Today I managed to work out one of the weirder bugs with the antidemotion features of my JAuthTools that has been an issue for a while now. I was actually looking at a Joomla! Bug at the time and this issue struck causing itself another problem of it’s own. In working out that something clicked and worked out why this was behaving the way it was.

Today was the submission of an assignment for uni. As I spent the majority of the weekend working on it I’m looking forward to not seeing it for a while. It must be said, however, that it has driven some testing into different systems and was one of the reasons for my SSO manager development so that part of the system was easily user configurable. So over the weekend I’ve managed to get the system to work whilst also progressing towards a release for JAuthTools 1.5.4 with the new SSO manager in it. At the moment Im still waiting on some feedback for the LDAP user plugin prior to release. At the moment I’m happy with it’s status so when I get some final feedback I’ll do what I hope will be my final release for 1.5 and start to look only at 1.6 stuff. Exciting!

I finally managed to get some work done on my SSO Manager, so now it handles service providers as well as publishing items. It also provides the forms for type C links as well as the service provider links for type B plugins. I also did some work with module so it will handle it as well. At the moment I also have two modules for SSO, one that handles displaying the forms and links as well as another to bootstrap SSO (called SSO helper). Still coming together and hopefully I’ll get a release sooner rather than later.

Spent another day with SALMON today rebuilding the parser to include the entirely new file format that we managed to create out of it. This one now produces the groups that Servers Alive creates so I can use those to automatically create pages and autofill in servers to it (which now works). So now it not only pulls in data about services and their status, it grabs the ‘pretty name’ as well as the actual item being measured, it also stores the result of that even though it doesn’t use it yet as well as the last time it was updated. I’ll put all of this together into something more useful but for now the parser works on the new file format. It now also autogenerates pages from the group and automatically adds the relevant services to those pages.

Other fun tasks for the day include more documentation (this time on docs.joomla.org), a blog post (community.joomla.org) and looking through my favourite forum maybe even writing a reply or two (forum.joomla.org). I did have a look at my SSO stuff though it was rather limited. Hopefully tomorrow I’ll have more time to spend on it and I’ll get it up and running with a proper installer so that I can get other people using it (yay!). I also noted today that our Human Resources department started to use the TIFF to PDF service that I offer at work after nobody touching it for a rather long period of time. Nice to see stuff used every so often and justify its existence.

So today I did some more work on my SSO system – though I had forgotten just how far I had progressed with it. Its still got a fair while to go yet mind you, but its done quite well. I added the ability for it to handle Type C plugins and the module now should also handle Type C plugins (if you don’t know what this means check out http://sammoffatt.com.au/jauthtools/SSO/Writing_an_SSO_Plugin). Next on my list is supporting service providers (Type B plugins) in both the front end and the back end of the system. I also managed to (again) bash out some more documentation, this time JAuthTools related. With any luck I’ll do some more tomorrow whilst I’m working on my SSO stuff.

The most productive task of today was writing up the documentation to JDiagnostic. JDiagnostic is my Swiss Army knife for stuff like authentication in Joomla!. The setup that I used was on the Mac OS X G5 workstation that I have at work with a cut down Safari window, Firefox in my Joomla! admin panel creating new articles and the Preview application to handle capturing screenshots. To make uploading to my server easier I used sshfs to connect to the folder I was after on the server which meant that saving files to the box was as simple as selecting the server on the file save dialog and waiting for the file to be uploaded. Thankfully the application saving the file is the only to block so I could take a screenshot in Preview and then type it up in Firefox and by the time that I went to add the image to the article the file had uploaded and was ready on the server to add to my new article. Preview is the stand out application which is a bundled application in OS X. It handles most image files easily and has a rather useful screen capture aspect built in. This tied with the menu bar at the top makes it really quick and simple to make screenshots – perhaps the easiest of all of the major platforms (Windows in my mind is second here because screen capture on Linux generally sucks except for the GIMP but it loses on usability). Another day of fun!!