I’ve been working today to convert a legacy app up to Joomla! level. As I progress with the app I get annoyed with the painful assumption that it makes that everything is in the right directory. It reminds me of PEAR that relies on people have it in their include path. In the world of shared hosting that might be something impossible: no ability to use PEAR to install packages and sometimes no ability to change the include path. Then you end up having apps breaking when either of these assumptions break. CiviCRM has this issue, amoungst others.

The day ended with acting principal (we go you, co-ordinator, principal, manager, director, CEO; some may be skipped, e.g. I have no co-ordinator) reaffirming that I should be working on backup systems as a priority, works for me. It was handed to him as a task to have this meeting and he left it until late in the afternoon on his last day. Obviously high on his priority list. Mind you it told me what I already knew, so it’s nice to be affirmed.

Later tonight I think it’s time to release JAuthTools now that everything is in order. More documentation to write.

Another day and a new feature request for SALMON. Today’s request was a request for a metaitem for if Servers Alive hasn’t updated the page for a while. There was another request for an audible alert though I managed to convince our network tech that SALMON wasn’t the place to build that sort of item. Tomorrow I’ll go through go and make sure to get rid of some of the cruft then release it out to the world.

Beyond the ordinary helpdesk style tasks, I also did some work on one of my smaller projects. I had forgotten that I had done some work on it elsewhere an after making all it changes and committing SVN gave me a resource out of date error. I update and feared that I’d have to fix up a conflict but everything merged properly where the different changes had been made. Worrying something else would break I refreshed the page and saw that it had actually fixed another issue that I had been ignoring temporarily.

I also spent a short amount of time making some changes to the Joonla! Trunk install file. Nothing major mind you but some slight changes.

All in all, happy with the day.

The most interesting aspect of today was getting the filesystem restores working for 1.6, something that due to the way that I put it together wasn’t that hard to put together. Now that it’s up and running I need to integrate it into the upgrade system and give it a work through, will be good fun.

Today was the first day back after not only a long weekend but the Joomla! Developer Summit. I’d left work early on Thursday to attend which meant that I missed the flooding of our helpdesk and server support. My desk had been taken over in part by some of the server support people. Thankfully my Mac was still mostly plugged so I got started with of and then progressed through the day to get everything else out of the way though my desk still looks like a bit of a war zone (as well as the rest of the branch).

Nothing much interesting today but I did get some work done with Ian MacLennan about an OpenID plugin for JAuthTools. Still not ready yet but getting there.

Beyond the incredibly boring and time consuming task of building an SQL query for a CSV dump of our grave records for yet another third party to put it in another system to be managed. Looks like fun.

Migration fun again with people wondering why their site didn’t migrate properly with core hacks, fun for the whole family. Another one with strange anomolies which appear to be due to a lack of the iconv extension (took a few posts to get that point including asking twice but all good fun). Hopefully removing those nasty core hacks and installing iconv will fix most things up.

Towards the end of the day I decided to try my hand at creating a new MySQL storage engine and I’m presently not doing too well. There is tonnes of pieces of documentation scattered over the web which are incredibly light on how the system actually works. I think I’ve managed to get a grasp on how the entire thing comes together and data gets transported though I’m yet to get a build process that works properly. I think I’ve done everything I need to get the item to work and with a bit of hacking of the configure file the system bulids though I’m stil not sure if its working properly (mind you its not failing at the compile stage so thats a start). Tomorrow I’ll see if what I’ve got at the moment will load into the MySQL instance I’ve built and progress from there.

The majority of today was spent driving around the place, first grabbing Wilco from the airport in Brisbane and later grabbing my sisters laptop from NextByte downtown. Nothing much excessively interesting though I do have an update from one of my LDAP user plugin testers that they’re having issues with things still so I’ll see if I can work that out in the next few days and not delay the JAuthTools 1.5.4 release too much longer.

After many weeks of development I spent the day testing the last of the packages for JAuthTools. There were a few files which had out of date manifest files or just the stock manifest that I copy around as a placeholder. Everything appears to be installing correctly so far so after I write some more documentation tomorrow or the next I’ll do a release.

Over the weekend I also completed some more work on JAuthTools finalizing the LDAP user plugin (hopefully) as well as creating a new module called “context login” to handle contextual situations such as Google Apps for Your Domain. I’ve retro fitted support onto both my advanced LDAP plugin as well as my advanced GMail plugin. Will need to write up how to use this new tool as well.

I also released JDiagnostic today with some updates to how I handle Active Directory with SSO better. In the next few days I’ll do some documentation on that as well.

Tomorrow marks the first arrival of The dev co-ord summit with Wilco. The rest of the gang arrive on Thursday. Should be a fun weekend ahead.

Migration is one of the more continual banes of my life and its one of the more interesting problems. Migration from Joomla! 1.0 to 1.5 combined many many changes. Not only did we alter the way the system worked in places in a somewhat drastic way incidentally removing features but we forced everyone for the first time to consider their encoding properly. A lot of sites were using the default latin1 encoding, even if they weren’t actually a Latin 1 language. Some sites used Latin 1 but some how in their database was actually UTF-8 characters every so often.

When I sit here and consider this, I realise that we really did make the right decision to almost force people to create new databases and new tables that were in UTF-8. It means that all of this ‘fun’ with encodings won’t happen unless we move from UTF-8 to UTF-16 or something similar in the future. Whilst its an incredible amount of pain right now, once we’re over this hump it is for the most part plain sailing.

The other interesting aspect of the process is the design of the migrator itself. When David Gal released the first migrator, it was basically a SQL dumping tool that was specifically designed to work with only the core tables that shipped with Joomla1 and to move those aross into the new site via an SQL dump. The design of the migrator that I built took a similar approach of making an SQL file that could be used to install items but it also addressed the concern that third party developers couldn’t extend the original version. My version of the migrator was heavily based around the concept of an ‘ETLPlugin’. An ETL plugin was a simple PHP class that at the bare minimum provided a descriptive display name, the name of the table in question and that was it. Presuming you had no changes in your database, writing a plugin was quite simply four lines of code:

class TableName_ETL extends ETLPlugin {
      function getName() { return "TableName ETL Plugin"; }
      function getAssociatedTable() { return 'tablename'; }
}

Thats it, thats an ETL plugin. The migrator has code that will find the columns in the table and build the appropriate SQL statement to get it into the system. You can add stuff like “getSQLPrologue” to prepend stuff before your rows such a “CREATE TABLE” statement.

As far as I can tell, the only vendor who actually made an ETL Plugin was DocMan. After complaining that they didn’t have a method of extending the tool to support themselves, they didn’t even bother to use it. Documentation on ETL plugins is in my Migrator RC1 OpenOffice.org (ODF ODP) presentation that was dated August 2007, that is around 5 months before we shipped 1.5 – and even now one year later I am yet to see any more plugins. Doing a Google search on “joomla 1.5 etl plugin” reveals links to the official documentation and copies there of, one link to the ODP file mentioned above with “Developing ETL plugins” in Google’s abstract and forum posts for Azrul, CommunityBuilder and a Real Estate component all mention “manually” migrating the users data from the database stating they don’t need an ETL plugin. For some reason they wish to make their users lives harder by forcing them to go outside of the system that I put together to make their lives easier. They could write an ETL plugin really easily and not have to worry about doing anything but shifting files manually (something mostly easily achieved with most file browser/FTP tools).

At the end of the day its sad to spend time building a system that will handle their needs almost automatically but still deliver flexibility if they wanted it and then have it not used and have them say to use a manual migration anyway. Perhaps I’ll write a post on how to make an ETL plugin (again).

For the last few days I’ve been wondering about getting XDebug up and running on my Mac with Leopard since it doesn’t ship natively. Now a few of the guides I saw suggested that I use the Komodo one that was already precompiled for Mac OS X, great! I wasn’t particularly in the mood to source compile something so I went off and downloaded it and then dumped it into the default PHP extension directory for Leopard (/usr/lib/php/extensions/no-debug-non-zts-20060613/), reconfigured php.ini to add an xdebug section and load the file and restarted Apache. No joy, the module wasn’t in Joomla!’s PHP Info page when it should be. I fired up Console and looked for Apache’s log files and found this charming looking message:

Failed loading /usr/lib/php/extensions/no-debug-non-zts-20060613/xdebug.so:  (null)

A quick Google resulted in a few blog posts and similar that basically had the same conclusion: compile from source. One blog post recommended to source compile but noted that the command line version didn’t work when the web server worked and another one pointed out that Komodo’s version was i386. This got me thinking and I did a quick check. Under Leopard, Apache (httpd) is a 64 bit application (check it out using Activity Monitor, enable the “Kind” column and it’ll say “Intel (64-bit)”) however if one runs “file” over the default PHP they will find its a universal binary compiled for PPC and i386, no x86_64.

So the solution to the dilemma of it not working properly is to build a universal binary with all of the platforms we care about, which is what I’ve done. I cross built the module for PPC, i386 and x86_64 individually and then used “lipo” to create a fat binary called “xdebug.dylib”. I then took this fat binary, dumped it into the extensions directory and loaded it using the “zend_extension” command. I restarted Apache and checked the logs for the error and found none. I then looked at PHP Info in Joomla! again and this time I xdebug was there. On the command line I ran “php -m” and noted that xdebug was now in the “Zend Modules” list. Unfortunately my old PPC G4 died (RIP) so I can’t test this on a PPC Leopard machine (my work one is still Tiger unfortunately). It should work mind you even though I’ve not tested it.

You can download the file with some install instructions from JoomlaCode: http://joomlacode.org/gf/download/frsrelease/9372/35384/xdebug-2_0_4.zip

In other news I spent some more time today looking over 1.6 and thinking about how I’m going to handle the file backup aspect of the system. I’m considering using a tarball style system to create the files and working through stuff like what options are required (exclusion filters? inclusion filters? exclude paths or include paths etc). I’ve written down some thoughts and I’ll probably continue working on it on Monday. For the weekend I have on my list getting some work done on my LDAP User plugin and releasing JAuthTools’ 1.5.4 release. I’ve got a test site arranged for myself so I’ll be able to play around with things which should be good. An exciting weekend ahead (at least for me anyway!).

Today I spent some time looking over the 1.6 backups code where I’d let it off a while ago and managed to get it working. There were a few really simple things wrong from when I’ve changed something in one place but not another but now its up to the point that its working properly so I’m happy. Annoyingly I keep using an old 1.5 instance to restore from so I’m going to have to do a new 1.6 instance and use that for my new dump files. Mind you I’m happy that this little side project has come together, once I get the file system level backups working properly (at the moment it does in-database table copies and SQL dumps/loads) I’ll be able to put it together into the installer subsystem and have full packages made of items before install. I’m still trying to work out what Joomla! upgrades look like yet, but I’ll figure it out sooner or later.