• Joopal and Drumla: not your usual mashup
• Application level replication in Joomla! using graphs

Firstly I’ll do a quick overview of the two main systems being promoted: Git and SVN. SVN, or Subversion, was developed as a newer version of CVS. Subversion has been referred as a “code wiki” which I feel is a great explanation. It keeps versions of files around for you and ensures you’ve got everything in hand. Subversion is a centralised source control system, so you need a central server to run it on for different people (you can also run it on your own machine as well but there is only one point of truth). Git is a distributed version control system where basically every working copy has a full copy of the tree. This is great for pure source projects and a few other sorts where having everything on disk isn’t too bad but doesn’t work well when you have larger repositories and files that you might want to share. Git doesn’t permit partial checkouts as well so you get the entire repository or nothing. This can work in your favour (you can do stuff like commits locally and do history checking locally) or against you (it might be a chunky repository). Git repositories typically are more compressed and smaller than SVN however Git’s Windows tools leave a lot to be desired. If you’re working with people who aren’t technical, Git can be painful and I’d suggest Subversion. Both are a learning curve but Subverion’s is easier and the centralised control is useful for most projects.

Provider A: GitHub
GitHub are one of the most popular Git hosting sites out there for open source projects as well as being a commercial hosting provider with “private” repositories. For people who are doing open source projects and interested in using Git, GitHub with its 300MB disk space (expandable for open source projects) and unlimited public repositories and collaborators is perhaps the most powerful option in the Git sphere. It is also all backed up as well, like most of the options, so you’ve got some peace of mind there. GitHub have personal and business branches offering different “private” hosting options starting at five private repositories consuming 600MB with one additional collaborator for $7 per month. The plans go somewhat incrementally up from there adding disk space, private repositories and private collaborators. GitHub has a wiki as an option as some point as well as a private/public pastebin service.

Check out GitHub’s pricing at http://github.com/plans

Provider B: Unfuddle
Unfuddle is something I’ve just come across after my presentation on a recommendation from someone at the conference. Unfuddle offers both Git and SVN support as well as a form of a wiki in what appears to be “Notebook pages”. It limits you to one active project but features RSS and iCal support as well as bug tracking, milestones and in the free version support for two people to collaborate. The free version offers 200MB and more expensive versions have file attachments, SSL and time tracking as well as more disk space, active and archived projects, people and unlimited “notebook pages”.

Check out Unfuddle’s pricing at http://unfuddle.com/about/tour/plans

Provider C: CVSDude
CVSDude are a much older group who offered initially CVS hosting but recently handle Subversion. They support Trac which a popular development support tool that integrates with Subversion providing milestone support, issue tracking and integration (e.g. you can close tickets from SVN) and a wiki. CVSDude appears to be slightly below par with GitHub with their cheapest plan offering 500MB of storage, one project and two users (as opposed to five projects and 600MB from GitHub). They also appear to offer Bugzilla, a popular bug tracking software (perhaps they didn’t like Trac’s version or have disabled it?) as well as DAV storage whatever that means. CVSDude does emphasise that they have better backup facilities than others offer plus the Trac/Bugzilla instance provides more functionality than GitHub does. CVSDude annoyingly hides a lot of information behind marketing so you need to do a lot of reading to work out what they’re really selling for each option.

Their overview page serves as an entry point for finding more information, check it out at http://cvsdude.com/hosting-products.html

Provider D: GForge Group (and JoomlaCode)
GForge Group appear to offer free one project per person hosting on their stack with a 75MB space offering. You can add up to five people to the project and it is a private project. GForge offers a wiki, mailing lists, forum, file release system, tracker and a few other tools as well. It has the interesting caveat that if you don’t log in for 30 days your project will be permanently deleted. GForge are selling a stand-alone product more than anything so they’re encouraging you to head that way with that however as an item it is an interesting. It looks like you can add to it however their store link didn’t appear to be working properly. JoomlaCode is powered by GForge AS and offers many of the same features (version control currently limited to SVN though GForge AS supports CVS and GIT amongst other things). JoomlaCode’s hosting is free for GPL non-commercial Joomla! related projects and is offered as a service to the community.

Check out http://gforge.com/gf/register/?action=ProjectAdd for more details.

Provider E: PixelNovel
PixelNovel is another host I’ve just seen today that offer a tool for Adobe Photoshop that integrates Subversion straight into the tool. This means that you don’t need to jump out of the system to handle it and it also does previews of the Photoshop files for you for when you’re going back in time. The standalone Photoshop plugin will work with seemingly any Subversion repository and costs around $60 per licence though it would appear you can pick up a free copy with a PixelNovel account which offers 100MB for nothing and goes up from there.

Check out their pricing and plans at http://pixelnovel.com/pricing

As with everything before you hand over cash, code or templates read the fine print. Though it doesn’t say it outright, PixelNovel for example will delete your account after two months of inactivity or lack of bill payment and GForge have similar albeit much more upfront text. Some services offer SLA’s on performance and uptime guarantees where as others don’t whilst some mention backups in a very definite time frame (I think CVSDude offers 10 minute backups) and others mention that they do it without many details. Some also offer more tools than the others and PixelNovel has some specialised tools targeted at designers particularly. As with everything the devil is in the detail so good luck checking things out and make a decision based on your own personal needs.

Currently ePrints lists authors by using the creator field of an ePrint. This is usually entered as the cited name of the author. If you have a small number of authors with different names this works reasonably well as the system can differentiate between people without much issues. The problem comes in when people are cited differently or have very similar names but are distinct people. Then it becomes bad.

One way of solving the problem is really quite ugly: you go through and change the cited author to make it look “neater”. In fact one of our library staff decided that they would do that greatly damaging the value of the data stored with in the system, tainting it horribly and creating more work for themselves. The cynic in me wonders that they’re just creating more work for themselves later to go back through and fix it all up. So there is a solution, non technical and pretty ordinary.

So our solution was to turn authors into a first class data set within ePrints. They’re not some unidentifiable free text field with no great purpose (there was a creator ID field but email address was suggested to be used for it) but in fact they are now their own entity which is cool.

We’ve done some different stuff with the author ID project at USQ that I’d like to share:

So an author has a name, they can be linked back to a user ID, they have an email address (both primary email address which should be used for contact and alternate addresses for aiding search), a biography and external identifiers. A cool feature of the primary email address is that when it is updated it automatically gets copied to the alternate email address list,  so you can easily update an email address and still retain the old one (you can still remove them from the alternate email address easily). The external identifiers section provides the ability to list useful identifiers for the author that mean something outside of ePrints. This could be stuff like a staff or student number that you record against the author to help with recognising your users. This is also a new field type developed called ‘Name-Value’ to represent name value pairs of data and provide an editing interface.

But the problem with authors is that they don’t stay the same: they’re dynamic. So they might move around an organisation, shift faculties, join a research centre or even leave the organisation. Recording this in the author record is more than possible but it starts to get messy – we start talking about ‘versions’ of the author but versions imply the wrong concept. A researcher might be working on papers in two different faculties, each paying for his time and each deserving of recognition. In our case we also need to use that information for reporting so that we can properly allocate funds based on research outcomes as well – for us it is important to know who paid for the research.

So to solve this problem I created a system called “author instances”. Author instances provide a subrecord of the authors that permits instances of specific data to be created and associated with the author. Then when an author creates a paper a link is created to this instance and not directly to the author. This allows the author record to contain information that is relatively static and data that might periodically change (such as department or even which institution someone is at) is then located in the author instance. As this isn’t ‘versioning’ per se it also works well for when a researcher is concurrently producing papers for different parts of the institution (or even for bodies external to the institution).

This screen depicts the edit view for an author instance.

The author instance is very light by default, it only contains the author that it is linked to, its display name and a preferred flag. The preferred flag is mutually exclusive within author instances – if one instance for an author sets the flag then it is cleared for all other authors. In the screen shot you will notice an untranslated field called “deptid”. This is another new metafield that has been developed called “Externalitemref” which is similar to the built in ItemRef field however it works on tables managed outside of ePrints. ExternalItemRef takes the params of a table name, a key field and a name field to operate in a similar manner to the ItemRef and link data back in.

The observant will notice that both the ExternalItemRef and the built-in ItemRef field also have a find entry button. This button has also been added to the ePrint item editing screen as well and triggers a popup window which allows easy searching for the particular item (user, author, author instance) that you are interested in.

The standard ePrints creator fields have a "Find Entry" button as well.

So as we can see there is now a “Find Entry” button that pops up with a new window allowing us to select an author instance for this author easily and not have to worry that we’re going to mistype their email address or similar.

This is just a quick introduction into some of the changes that the Author ID system which USQ has developed provides. Ideally we’d love to feed this back to the ePrints core so that everyone can use this, or a similar system, in a great way. If you want to read more you can check out the “Author ID Notes” document on my knowledge base download site.

Work again is fun as always. We’re working on solving various election problems and getting envelopes. Our printers came back stating that they didn’t want to do Code 128 for the barcodes because its too hard preferring Code 39 instead. As the day progressed it turned out that we’re going to use Code 128 more for space issues because we’ve got that much junk going out on the envelopes. This saga has been occurring over a few days, I really want to get a few proofs done before we actually get the batch done from them first to ensure that everything is good but we’ll see. Testing on the system we’re going to be using to track the election on has also begun to ensure that the system can in fact handle items properly, we had a word of warning from one of the newer guys from the regions that it had issues and that was with nowhere near the same workload. So we’re hoping we can test the system out with a few VM’s emulating the system for multiple data entry. There have been issues with the system in question in the past (in fact the system is being replaced)  so hopefully this won’t be a time when it does have issues otherwise we’re going to have large numbers of problems.

The bulk of my day was spent working on the restore framework for Joomla! 1.6, or more accurately the new JDataLoad system and the JLoaderSql adapter. The data load system, as its name suggests, loads data into the Joomla! database from a data source. In this case I’m looking at the SQL files which in my sample data is actually one of the 1.5 sites that I’m an administrator over at work. Its relatively small in the grand scheme of things with only a few meg and around 9000 queries. So far its been sufficient to find a few issues, one being dropping a table before a task yield which caused a missing table error from J! (put in a simple patch for that, if the last query was a drop, go to the next query in the hope that its a create) and another was a minor typo error which caused some multiline strings to be processed incorrectly when they were on a yield boundary again as well. But all in all its working well even importing data faster than MySQL Query Browser was (to its fairness it highlights each query as it goes) in my test runs. Its now committed to trunk and when I get a chance I’ll write up something about it and put it somewhere

Extra fun today came from trying to write up business cases for the projects I want to work on in the next year or so until I have to write it up again (fun, yeah!). Initially my boss (who is great) thought of me and tried to convert the files from Excel into a more Open format so that I can get at it on my Mac and Linux box. Somewhere during the conversion however the fields got trimmed and data lost, so I offered to edit the document in Excel directly using our Windows only document management system logging in via our Citrix services. The system, OpenText’s DM, isn’t too bad for the most part and does the job well and today I found no fault with it. Today was the day when Microsoft’s tools decided they wanted to misbehave.

Earlier in the week my boss had emailed me a doclink to the document stored in DM. A doclink is a small text file with the document number in it which basically triggers the system to load the specified document, something that usually works quite well. However Outlook, due to various configuration changes, decided that it didn’t want to start for some reason even though earlier in the week it was working perfectly fine. After Outlook repeatedly informing me it wanted to recreate my profile and then informing me that it couldn’t contact my Exchange server and offering to allow me to work offline which consequently failed due to a lack of a profile, I ended up using Mac OS X’s built in “Mail” application (yes, the email application is called “Mail”) to get at my email to find the document number and open it. Usefully enough this is done through the IMAP interface on Exchange and worked well and doesn’t suffer from some of the other issues that the Outlook clients have, such as the address book caching which caches the old Lotus Notes addresses instead of their newer Exchange ones resulting in emails going to the wrong place. Yay Outlook! Suffice to say I found the file and made the changes that I needed to before accosting our Exchange administrator who had returned to see if he could fix the issue – which he did mind you after some trial and error.

The last little item I looked into was building a system to version the content from Joomla! back into our document management system. The new document management officer assures me that we can do it and has even proposed a nifty way of importing the data into the system. It looks like one of the products we have, KoFax, will help us by allow us to generate XML files which specify the documents that we’re creating and the different versions. If we can get this to work it will be really awesome as it’ll mean that our website is in part integrating back into our document management system without hacking into the database! I’ve still got to build it and work out where we want to target the extension, but suffice to say its on my project list for next year.

The Kerberos keys that I had asked to be remade were ready for me by the time I got there. It took a bit of time to rebuild the different keytab files to support the vhost environment (need to merge the respective keytab files) but once that was done everything was working. Well, mostly working. Firefox on my Mac worked fine, Firefox on the Windows desktops I tried worked when they were configured (see http://grolmsnet.de/kerbtut/firefox.html for information on what you need to do to get Firefox to do negotiate), IE on most of the desktops worked fine however some installations weren’t getting SSO, all of the Citrix servers seem not to pass through authentication (they end up going in a weird loop where IE appears to keep loading the page) and Safari on my Mac doesn’t seem to want to play the game either. Perhaps I’ll sort that out over the next week or so but that consumed a reasonable amount of time going through and checking different IE versions and if they worked. The only machine not to play the game seems to be Firefox on my Linux desktop (it should be working) so I’ll have a look at the ones that don’t work and why they don’t want to work. For the Windows boxes I have the feeling that the Netware client is causing issues (which would explain Citrix) so hopefully when our network eradicates Novell we’ll be fine.

And that leads us to the afternoon’s fun of building Joomla! 1.6′s backup system. I’ve managed to get the system to export the sample database, reimport it and then delete the files afterwards so I’ve moved onto much larger goals. I’ve taken one of our internal websites and I’m trying to get it to important. Suffice to say that it has enough data to cause an issue with the system. For data loading I’m using a heavily modified version of Alexey Ozerov’s “BigDump” script, which has been used in the past in a less modified form for the Joomla! 1.5 migrator. It is slowly being converted to use the new Tasks system in 1.6 which is another concept borrowed from the 1.0 migrator. The Tasks system in 1.6 has two items: a task set which is a container for individual tasks. So considering backups, one task set might be a full backup run of the site with individual tasks being an SQL backup, a file backup (tar archive perhaps?) and maybe copying that to a remote FTP site or similar. So the one task set would have an “SQL backup” task and a “file backup” task. Extension package installation may do a similar item as well splitting the install into different parts.

A new part of this is the data load system that provides functionality to read and load data files, at the moment only supporting SQL but I’m hoping I’ll be able to create a CSV one as well some luck, again probably reusing Alexey’s code in part here as well. I’m mostly through building parts of this system though I’m experiencing a strange issue with my sample data (hence why the updates haven’t been committed to J!’s SVN repository today) where it loads the file up through to almost 2000 queries and seems to stop suddenly. I’m not quite sure whats going on but I’m happy enough that the task system is picking up and storing values for it to progress as far as it does.

Another successful day spent on my Mac as well, NetBeans doesn’t seem to want to look at my project any more crashing instead of loading it which is disappointing but I’ll work that out another day. And now its time to enjoy some Dawn of War.

Today is Melbourne Cup day, being the first Tuesday of November, so we had a luncheon of sorts and a drawing for the horses. Didn’t win, the food was good, I’m $10 poorer and such is life.

I’ve been spending more time at work using my Mac as a primary machine. Since I’ve moved to Exchange from Domino (or Outlook from Notes), I’ve gotten Evolution on Linux mostly working (with the exception that it doesn’t automatically look up names for emails which is tedious) and Apple’s Mail and Address Book both playing nicely with Exchange. I do miss the fact that I had Notes on my Linux desktop and things mostly worked albeit slowly and consuming large amounts of memory, but it worked with all of the features available normally. Mail’s ability to due autocompletion is what is drawing me back to it as a client, which when you start writing emails is actually more useful than you would think. Its still not up to par with the Notes autocomplete which was quite cool and a lot more advanced than either Mail’s or Outlook’s (I get Outlook via Citrix).

I’ve also been trying out NetBean’s PHP Early Access through a nightly build (has the ability to create PHP projects from existing sources) and I’m impressed with it. I tried it out because I wanted to try out debugging with my PHP instance and the dated version of Eclipse I had (3.2) seems to have issues – more than likely my fault – and I don’t want to waste time on trying to fix something. NetBean’s installed and worked almost instantly, however it took me a while to find where I could change the params to get J! to route items properly. I managed to work out the bug that I was having without too much issue. I knew what it was but not where it was: turned out to be exactly what I thought, an assignment operator used instead of the append operator. The Subversion support seems to be a bit off and doesn’t work yet, so I’m not quite ready to ditch Eclipse yet – but I’ll try with later versions to see what I get.

I had a chat with the principal (we have principal, manager, director, CEO as our chain of command) about the projects that I’m doing and the ones I’m interested in so I’ll have to do some paperwork and business cases for the new projects and justify items. We’ve recently got a new manager who is trying to find where everything is so part of this is explaining everything so that he can get a grasp of the way the system works.

Then I spent the majority of the afternoon with one of the ITS guys working through how our Citrix boxes work with Flex profiles and the mandatory profiles filling in the gaps in his knowledge and how different parts of the system and why items might break or behave in a particular way. I think he’s worked out how it works and he’s even figured out why a few issues are happening. So nothing exciting but useful.

And finally I had fun with Kerberos. I built the Kerberos module on the SLES10 server, installed it, restarted Apache and tried to get it to work. On my Mac both Safari and Firefox requested a username and password instead of using a Kerberos token and IE6 in my Citrix session seemed to just go in a weird infinite loop. I slowly worked through my entire Kerberos configuration on the server until I got to looking at the keys. It turns out that the keys were created with the wrong virtual host name for the server which is causing the issues. The keys for the real server name actually worked fine when I got around to testing them which proves that everything will work once I get the keys. The last part is a fix to the Citrix system which for some reason think that the intranet site is actually on the internet, but I’m assured that this should be easy to achieve. Getting Kerberos up and running was pretty easy ignoring the faulty keys compared with some of the nightmares I’ve had getting items to play nicely together. I’ll probably add something to my guide (http://sammoffatt.com.au/jauthtools/Kerberos) on it, to help with items.

Who knows, I may have even figured this Kerberos thing out!

http://forums.cubecart.com/index.php?showtopic=33714&pid=151145&st=0&#entry151145

Their server went down which means you couldn’t (temporarily) update your site. More of a disservice. Similar to Microsoft Windows Genuine Advantage issues legitimate people have had with that “service”.

I’m also looking at doing some work on the JAuthTools for 1.5 to improve support. There appears to be some issues so I want to get it back up and running as well as porting/merging some of the features from the LDAP SSI into 1.5 and the LDAP Authentication plugin. I’ll probably also do some update to docs on the wiki as well to reflect the new features.

Things on my todo list after I clean up my released JAuthTools plugins:

• Backlink Manager
• JAuthTools Manager
• JDiagnostics for 1.5

First up for today is a personal thing, I completed a rather largish Uni assignment today which reminded me of all of the pains that come with C++, but to follow that I returned to working on my filesystem in C, which is just more pain. I got a quick response back, and almost full marks (96%) so I’m happy for all of the time I put in to get it done and how its probably far more complicated than anything else that will be submitted (it used Boost Signals and a whole heap of other things that I don’t think will ever be taught in the subject for a long time). But hey, thats just Uni!

Today I finally managed to get Pentaho, some business intelligence (BI) software, to play nicely with Novell eDirectory’s LDAP interface. I must have missed the option, but Pentaho doesn’t seem to accept anonymous binding to the LDAP server, which means I need to bind as a user. By default our users funnily enough have less access than the anonymous account (which is actually a proxy account with full browse permissions). The solution was simple enough: we shunted our dummy Pentaho user into the same group as the anonymous proxy account and everything worked. So I’ve now got Pentaho using LDAP for authentication (yay!) and a MySQL database to get its role/group permissions. Funnily enough when its all said and done the documentation is pretty close to the mark.

But once I had that I don’t have an ability to manage the groups/roles within Pentaho, so I end up having to write some small PHP to manage that. Luckily I worked on a project a while back that I called “Joomla! Central Management for Users” which basically connected directly to MySQL databases of Joomla! installs and altered the users. I had originally built it with a plugin infrastructure in mind so that I could plug other stuff into it later. Starting this morning it only had a ‘connector’ for Joomla! 1.0 via MySQL and LDAP, now it has one for the Pentaho security tables. This means I can easily copy users from LDAP or Joomla! into Pentaho without too much issues and has a debugged user interface already. But wait theres more!

When I was originally developing the tool I wrote a query language for it. See, SQL is a great language for databases, but its a bit hard to apply in situations where you don’t quite need all of that power. So I wrote my own query language. Its quite simple it can validate simple attributes and allows for set operations within “Sites” (a site is a container for users and groups). So for example I want to see all of the users who are on our web site but not in our LDAP directory:
existsin “Web Sites” and not existsin “LDAP”

Primitive sure, but it because writing a large SQL expression for something simple. I hope to expand on it, but it already does what it needs to do for the time being.

So I’ve covered query languages, LDAP and BI! All I need now is the filesystem news. Today there was a whole heap of fan fare on Slashdot about the ZFS news from Apple, whilst thats cool and all (especially since I don’t mind Apple’s UI), I personally have my own filesystem that I’ve gotten back into to do some work on. It also happens to be a Uni assignment due on Friday! So I’ll be back to working on that and hopefully I’ll have it to a nice stage that I can do some lightening talks at linux.conf.au!

Consider an all Microsoft shop who had never considered Linux before but wanted to virtualise their products. The best product to do this from what I can tell is the VMWare ESX platform which is really a small Linux hypervisor and then their own management tools underneath. The aim is that the end user never has to see the Linux back end they only see the graphical tools for Windows like Virtual Control Centre or the web based interfaces.  This leads me to the funny thought that a pure MS shop that had perhaps sworn off Linux might be deploying ESX because it is the best option out there (MS hasn’t brought out their solution yet, thats waiting on their Longhorn Server product), an entire Microsoft world virtualized under Linux.

It makes you wonder where the world is heading.

So the solution from the above as to add a redirect to another virtual host on the server:

RedirectMatch permanent ^/error http://pasamio.com/error

Another solution is to rewrite the ErrorDocument to make it return strings instead of page: 

ErrorDocument 400 "400 HTTP BAD REQUEST"
ErrorDocument 401 "401 HTTP UNAUTHORIZED"
ErrorDocument 403 "403 HTTP FORBIDDEN"
ErrorDocument 404 "404 HTTP NOT FOUND"
ErrorDocument 405 "405 HTTP METHOD NOT ALLOWED"
ErrorDocument 408 "408 HTTP REQUEST TIME OUT"
ErrorDocument 410 "410 HTTP GONE"
ErrorDocument 411 "411 HTTP LENGTH REQUIRED"
ErrorDocument 412 "412 HTTP PRECONDITION FAILED"
ErrorDocument 413 "413 HTTP REQUEST ENTITY TOO LARGE"
ErrorDocument 414 "414 HTTP REQUEST URI TOO LARGE"
ErrorDocument 415 "415 HTTP SERVICE UNAVAILABLE"
ErrorDocument 500 "500 HTTP INTERNAL SERVER ERROR"
ErrorDocument 501 "501 HTTP NOT IMPLEMENTED"
ErrorDocument 502 "502 HTTP BAD GATEWAY"
ErrorDocument 503 "503 HTTP SERVICE UNAVAILABLE"
ErrorDocument 506 "506 HTTP VARIANT ALSO VARIES" 

This also fixes the problems, though in a few more lines. This is perhaps the better solution as it wipes out the old ErrorDocument directives that were giving us some troubles and returns a straight string. The next simplest solution is of course to move the Subversion “SVNParentPath” from the root to its own path underneath the root, which is the solution that most people seem to go in for doing.

The new tool is based around two concepts: an ETL plugin that defines a transformation that creates an SQL statement. For the most part these are extraction transformation (e.g. it comes from a database table), however there exists an ETL plugin (the configuration one) that extracts settings from the 1.0 configuration.php file and puts them into different parts of the 1.5 database. An ETL plugin is at the core quite simple and can be successfully implemented in as little as five lines of PHP code that extend the base ETLPlugin class. Those lines basically define the name of the table that the plugin transforms, a friendly name and a few brackets for good measure. This simply extracts the contents of the table without any modification on the fly. An ETL Plugin can also be more complicated with easy ways to skip table columns, rename table columns or alter the value of a column in a row easily (hence the transformation part). This is used to alter some tables as they are exported from 1.0 instead of attempting this in 1.5 as we’re already examining each row to generate the SQL file.

The majority of the work of an ETL Plugin is handled by the ‘doTransformation’ function which generates the SQL statement. For most plugins this function can safely be inherited by the subclass and other functions (which control the alteration of column names or values) should be updated to add relevant functionality. The ‘doTransformation’ function fits into the migrators task system and is executed as per the values of the present task (table name, and current iteration).

There is another form of plugin, called an SQL Plugin. The SQL Plugin is really simply a text file with SQL in it which is designed so that third party developers can place ‘CREATE TABLE’ commands in the SQL file before their ETL plugins fire and output INSERT statements that are going to be expecting a table. These plugins are stored in the ‘tables’ folder of the migrator, and the ETL plugins are stored in the ‘plugins’ folder. Existence of a file in this directory will mean its inclusion in the migration. There is no published or unpublished state since this should be a mostly one off event.

The task system is used by the migrator to prevent the system from timing out by completing excessive commands against the system. The task system works by asking an ETL plugin how many rows it has to transform and stores this in a table. Once it has done that it also collects all of the SQL plugins and places this into the new migrations SQL file. After this is all done the task handler is executed and starts the ETL for the tables. The task system works through each ETL plugin, calling the ‘doTransformation’ plugin. When the task system detects that it is getting to a set threshold of the PHP timeout it will pause the migration, update the present task and end output telling the browser (via JS) to load up the task handler again. It completes this for as many time is as necessary until there are no tasks left in the system and the SQL file is complete.

So now we have our SQL file nicely generated from the migrator and now we need to put it into 1.5 so that we can use all of the really cool features. One thing to note is that at present the behaviour of the migrator is to strip the prefix of the existing site and replace it with jos_ – I made this decision as a documentation reduction and simplification feature (so that the last step in the 1.5 install, where the migration is triggered from, would always be ‘jos_’, but I might change this as it appears to be causing confusion). This doesn’t mean that the 1.5 table has to have the ‘jos_’ prefix, but you do need to tell the migrator that this is the prefix. This leads me to the next step in the migration chain: a 1.5 install.

The 1.5 install completes normally until the final screen, the main configuration screen. Until this point the installation is the same for a migration or a blank Joomla! install. However there are some things to fill out on the final screen. This screen asks you for your site name, the admin email and password, and finally if you want to do a migration load or a sample install. We’re doing a migration so we check that box and enter the information. Remember the old prefix for the migration is always jos_ so put that in there. The encoding is going to be that of your old database. Your old database might be UTF-8 already so select that, if you’re from most English speaking countries the default should be fine, and for other parts of the world you will have to select encoding relevant to your part of the world. This is used by the migrator to convert the file from that format into UTF-8 (or do nothing if its already UTF-8!) using the servers ‘iconv’ function. IF you don’t have iconv installed, you will have to convert the SQL file into UTF-8 before you feed it to Joomla! and in which case you can set the encoding for this field to be UTF-8. For Linux users iconv can easily be installed on a local machine and worked from there. Windows users have various options (including a small Java application I have mentioned in one of my previous posts) to convert a file into UTF-8.

The last few options deal with Joomla! getting the SQL file and if its a migration file. For small files its easy to just upload via the HTTP upload interface, but for larger files that are over PHP’s upload limit, the files can be uploaded to the ‘installation/sql/migration/’ directory and call the file ‘migrate.sql’ so the installer can find it. You will need to tick the “I have already uploaded” check box as well as the “This is a 1.0 migration script” box. The HTTP upload option can alternatively be used. Keep in mind here that the SQL file will be altered by the migration system no matter what option you choose. So even if you have to FTP up the file, you will have to either copy over the file or reupload if the migration fails just as if it was a HTTP upload.

The next step in the chain involves a few changes made to the migration file. These changes involve rewriting some files, and altering the menu and modules table to insert their data into a temporary migration table instead of inserting them into the real table. This alters the file which is then fed into the data load script. The data load script then bulk imports all of the information into the database, again with checks to ensure that the system doesn’t time out. The script is a modified version of the ‘BigDump’ script written by Alexey Ozerov (available here: http://www.ozerov.de/bigdump.php) that handles the import for us (and saved me having to write it out!). From here the migration system performs some magic. It rewrites the menu’s to match 1.5′s new style, it grabs and imports the modules and it also does various bulk SQL updates to change things. What it doesn’t grab (at any point) is site URL’s that will have changed in either URL fields or in content items. From here the migration is complete and the site is ready to run.

At this point there are a few final things to note. There is a new menu module created and published as a part of the install (this is in addition to your old module). All of the other modules in your old site are transferred even if they are third party and they’re set to be unpublished. You can install 1.5 version of the modules (or the 1.0 versions in legacy mode) easily and it won’t alter the database entries, so modules can be easily migrated. Third party components that wrote ETL plugins will require a special 1.5 installer that doesn’t attempt to create database tables (they’ll already be there) to install the files and relevant database entries.

With regards to migrator status, I’m planning on releasing an update for the migrator this Friday, I’m attempting to resolve a final bug with duplicate rows being created in the SQL file. I’m not quite sure what is causing this but its my last bug before I release RC2 of the Migrator on Friday (or hopefully tomorrow GMT+10!). I’m also going to push through some changes to the 1.5 side as well to ensure that certain conditions are met properly, so stay tuned for that (I’ll write a short blog post on that when I release it, so set your feed aggregators up to Joomla! to stay current).

A final hint for third party developers, if you’re currently using a configuration file outside of the database, with 1.5 you might consider using the JRegistry functions and migrating your configuration file into the components table. There are some nifty lines of code that allow you to create a configuration object quickly using this method (very cool) instead of having to create your own configuration file. An ETL plugin, similar to the configuration one can allow you to make the transfer.