A friend of my is running a Drupal 6 website on a Combell webhosting. When he wanted to access the manager application of Combell (which is located in http://www.yourdomainname.com/manager), he was getting the HTTP 403 error.

I fixed this issue by adding the following line in the .htaccess file:


RewriteCond %{REQUEST_URI} !^manager/$


Generating backup

Each month, I take a full backup of my webfolder at http://www.jochenhebbrecht.be. It contains all webdata, as also a seperate SQL file for each of my databases. My host (http://www.vivior.net) offers me CPanel, which is giving me a lot of oppurtunities to do funky things. Up until today, I always manually logged in to CPanel, browsed to the Full backup wizard and started the job.

But today, I configured a cron job which does this job for me.

• Login to CPanel
• Go to the File Manager app and upload the following file - full_backup.sh - into your home directory:
  

  #!/bin/bash
  curl --user ##USERNAME##:##PASSWORD## https://##DOMAIN_NAME##/frontend/paper_lantern/backup/wizard-dofullbackup.html --data-binary $'------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="dest"\r\n\r\nhomedir\r\n------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="email_radio"\r\n\r\n1\r\n------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="server"\r\n\r\n\r\n------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="user"\r\n\r\##USERNAME##\r\n------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="pass"\r\n\r\n##PASSWORD##\r\n------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="port"\r\n\r\n\r\n------WebKitFormBoundaryGww39ElKNvLB43AS\r\nContent-Disposition: form-data; name="rdir"\r\n\r\n\r\n------WebKitFormBoundaryGww39ElKNvLB43AS--\r\n' --compressed

• Provide the file execute permissions
• Now go to the Cron jobs app and add the following line. It will create a full backup the first day of the month:
  

  0 0 1 * *   sh ~/full_backup.sh

• A full backup is now generated to your homedir.

Automatic download of full backup

I'm running Drupal 7 on a local Ubuntu installation. I've created a simpel module which automatically downloads the file at the 2 day of the month (as the 1th day, the backup is generated). This is the script which you can reuse:

<?php
function ftp_downloader_cron() {
	if (date('d') == '2') {
		// Configuration
		$ftp_server = '##FTP_SERVER##';
		$ftp_user_name = '##FTP_USER##';
		$ftp_user_pass = '##FTP_PASSWORD##';
 
		// Set up basic connection
		$conn_id = ftp_connect($ftp_server);
 
		// Login with username and password
		$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass);
 
		// Get all files in FTP home dir
		$files = ftp_nlist($conn_id, '.');
 
		// Get name of backup file
		foreach($files as $file) {
			if (preg_match("/^backup\-.*jochenhebbrecht\.tar\.gz/", $file)) {
				watchdog("jochus_ftp_downloader", "Found file: %file", $variables = array('%file' => $file), $severity = WATCHDOG_NOTICE, $link = NULL);
 
				// Download file
				if (ftp_get($conn_id, "##SERVER_LOCAL_PATH##/$file", $file, FTP_BINARY)) {
					watchdog("jochus_ftp_downloader", "Successfully downloaded file", $variables = array(), $severity = WATCHDOG_NOTICE, $link = NULL);
				} else {
					watchdog("jochus_ftp_downloader", "File could not be downloaded", $variables = array(), $severity = WATCHDOG_ERROR, $link = NULL);
				}
 
			}
		}
 
		// Close the connection
		ftp_close($conn_id);
	)
}

This tutorial really helped me a lot configuring Apache 2 and JBoss 4.2.4.GA on my Mac OS X (10.7 - Lion) installation. Just follow the steps and you will do fine ;-) ...

I'm always forgetting this key combination, so I'm gonna put this for once and for all on my blog :-) ...

Current situation

- Mac OS X 10.7
- VMWare Fusion with VM Installation "Windows 7"

The problem

I want to send a backslash (\) to the VM Installation. Normally, on Mac OS X you use the combination: alt + shift + /. But this doesn't work ...

The solution(s)

• Use the alt at the right of your keyboard and the < character: alt (right) + <

... OR ...

• Go into the VMWare Fusion setting and provide a Keyboard Mapping

I recently installed iTerm2 on my Mac, and I must say: "WOW! What a difference!"

The most popular features for me are:

Split Panes

Divide a tab up into multiple panes, each one of which shows a different session. You can slice vertically and horizontally and create any number of panes in any imaginable arrangement.

Search

iTerm2 comes with a robust find-on-page feature. The UI stays out of the way. All matches are immediately highlighted. Even regular expression support is offered!

Mouseless copy

Use the Find feature to begin searching for text. Press tab to expand the selection to the right or shift-tab to expand the selection to the left. Option-enter pastes the current match.

Configurability

Map any key to any function. Assign separate functions to each option key--or even remap all the modifier keys. You can customize iTerm2's appearance to suit your needs: enable transparency, background blur, background images, and much more.

Unixyness

Coming from a Unix world? You'll feel at home with focus follows mouse, copy on select, middle button paste, and keyboard shortcuts to avoid mousing.

Growl Support

You can choose to receive Growl notifications of activity, bells, and more. Feel free to let a long job run in the background, secure in the knowledge that you'll know when it's done.


You can find more information on: http://www.iterm2.com/#/section/home

Well, I had the possibility to "taste" from both worlds. I started at Dolmen (which is now known as RealDolmen). I got a specialized training in all kinds of Java frameworks. After 3.5 years, I was getting really pro in some of the JBoss frameworks (Seam, Hibernate, Richfaces ...). Also, EJB and JSF are frameworks which have almost no secrets left for me.

After RealDolmen, I started at Pigeon Paradise. A company very close to home, which gave me the possibility to have more free time to learn more things on my own. I had the possibility to play with Drupal at PIPA. I was really looking forward to this new technology. After a 1.5 year, I got specialized in Drupal 6. I've created a lot of custom modules, together with 2 other Drupal developers. I do have a lot of experience in working in a team on Drupal projects.

So, as you can read, I had experience with both worlds. I would like to compare the 2 "worlds" in this blogpost.

Drupal

The power of a scripting language

The main advantage I see on Drupal is actually an advantage of PHP. It's a scripting language (Java needs to be compiled to a binary file, afterwards, you can run that binary file in a JVM). When you develop in PHP/Drupal, you don't need to compile your source code. And most of all, it's very, very easy to just do whatever you want to do.
For example:

$content = '';
$result = db_query("SELECT foo FROM bar");
while ($row = db_fetch_object($result)) {
    $content .= theme("list-item", $row->foo);
}
 
return theme("list", $content);

So what you can see here, is a wrap of a database query, some (simple) business logic and a call to a theming function. It's really possible to program amazingly fast. If you made a typo, you fix it, save the source file, push F5 in your browser and tadaaaaa ... the new result.

Design pattern: Model-View-Controller

In my opinion, Drupal has an implementation of the MVC pattern. The hook_menu() is actually the controller and delegating all the requests to the correct page callback. In the page callback, you can perform some business logic and you can ask the theming function to wrap your model in a pretty jacket.


But there are things I'm missing in Drupal, and ... which Java is having.

Java

Build/release management tool

This issue came up because Java has a buildtool like Maven or Ant. Drupal has Drush. Drush is nice, but it needs some "team" features to control Drupal projects. It's not simple to control the lifecycles of custom Drupal modules, or the core of Drupal. I noticed D.O. has some kind of releasemanagement, but I wasn't able to set it up for our company. You cannot easily create branches, tags, trunk, ... You need a whole custom setup to achieve this. I once gave a presentation about this for drupal.be. Please check some files here: http://www.jochus.be/site/files/. It's hard stuff, because the core has his lifecycle, but contrib and custom Drupal modules have them too. You must keep them seperated.

Integration with IDE

Eclipse (or IntelliJ, or whatever Java IDE) and Java just match... They match great. There are so many nice plugin to integrate (such as JBoss Tools for JBoss frameworks), which can make development so easy. It actually makes programming very fun! In Drupal, I was missing some great plugins. I also worked with Eclipse (and PHPDesigner), but the IDE isn't just integrated with your project. If you want to debug your PHP application, it takes a while to set up the server as client side. In Java (well, actually ... Jboss), this debugging is going very easy, especially with Eclipse.

GUI library framework

Well, I have been using Richfaces a lot. I must say: I really miss this framework in Drupal :-( ...

RichFaces is an open source Ajax-enabled component library for JavaServer Faces, hosted by JBoss. It allows easy integration of Ajax capabilities into enterprise application development. RichFaces is more than just a component library for JavaServer Faces. It adds:

And of the things not mentioned above: the UI components WORK in different browsers. Drupal is still missing this GUI layer to easily create some pages. I'm not interested in writing custom HTML elements, nor in moving an input element 5 pixels to the left in Chrome, but 5 pixels to the right in IE ... No, I just want a < rich:tree > or < rich:datatable >. And this component should work in whatever browser you're trying to open it!

Object oriented programming

Drupal 6 isn't written in OO. I've noticed Drupal 7 got some improvements on this issue, but still ... there's a hugh lack of OO design. I miss this a lot. OO has following advantages according to me:

3-tier architecture

For me, the greatest "pain-in-the-ass" in Drupal. I do think you need to separate database calls and service calls. In a certain project of my previous company, I noticed the same query turning up 3 times in 3 different files, but they were actually doing the same :-/ ... I do think you should create DAO's. Combine all functions which are related to a specific domain object and group all queries together into one single interface. Next, call the function on this interface from your service layer. And let your service layer call your view layer. The link service/view layer is kind of present in Drupal, but Java just beats Drupal by using JPA (Hibernate) / EJB / Seam / JSF and Richfaces.

Amount of stable frameworks

In Java, I think there are 2 famous frameworks to work with: Spring projects and Seam projects. And that's it. There's a lot of information about these frameworks on the internet. And some really large communities.
In Drupal - you are looking for a specific kind of framework, you can find a lot of "frameworks". They are called contrib modules. For a specific kind of functionality, you can have 4 or 5 contrib modules. But none of them actually does what it has to do. And the most painful thing: some contrib modules are just placed online, but there's no working community behind the product any longer. So if you have some issues/problems/questions, nobody can help you out.

Conclusion

For me, Drupal isn't ready to handle real business applications. It's a good product, but I would only recommend it for really simple projects. To summarize Drupal: get input data from the user, store it in the database and retrieve it afterwards. I sometimes miss the complexity of programming in Drupal. Like BPM, a service bus layer, SOA, ORM, ...
I also think I didn't receive much time/space to discover all those things as a Drupal developer (or to start implement them - if you don't have it, you can try to create it) in my previous company. I cannot blame them: they are not an IT company. Their core business is selling pigeons, not creating a Drupal product and all other kinds of tools to work with it. If I want that, I should start at Acquia :-). I do think Drupal can achieve that level (which is Java having now) after a lot of effort. But it's gonna need time and coordination. And a lot of patience ...

Anyways, after 1.5 year working real hard on Drupal, I had to give up. I was getting too much frustrated in the way of working. It might have been my previous company's way of working, I don't know. But I knew I was missing Java. And after 1.5 years, I switched back to Java. I'm now programming for almost 4 weeks back in Java, and believe me: I HAVE BEEN MISSING THIS A LOT!

A long time ago, I made a blogpost about a solution for Drupal and a MASTER-MASTER replication setup: http://www.jochenhebbrecht.be/site/2011-01-13/drupal/drupal-master-mast…. At that moment of writing, I was very happy with that solution. But after some months of experience, I must say, I cannot agree any longer with my own solution. Some people were commenting that blogpost, and I really wanted to post my point of view of a new possible architecture. I definitely don't want to spread around a bad architecture setup. The new environment holds some improvements (which I cover at the bottom of this blogpost). To be honest, I must say that I only implemented the memcache improvement of the new architecture. There was no time/budget in my previous company for the other improvements, but I strongly believe those extra improvements are the way to handle a MASTER-MASTER replication setup.

Why is the old architecture so bad?

In the matter of fact, we were having too much issues with the old architecture. We were losing time and the architecture was just unstable. I'll explain why:

• The caching mechanism was initially configured to use the MySQL databases. So all CRUD operations on caching tables were executed towards the MySQL databases. Each web instance was having his own caching mechanism - we didn’t replicate the cache tables (replicate-ignore-table setting in /etc/mysql/my.cnf). The tables were handling string keys as primary keys, so it would be impossible to avoid conflicts. As a small reminder: you can only set even or odd integer numbers per MySQL instance to autoincrement ID numbers. So we had to switch to a mechanism where both instances were having their own cache tables, to avoid duplicate SQL entries.
• However, Drupal is writing so many things to the cache tables, the MySQL BIN log files were really getting overloaded. As I can remember, we had every 7 or 8 minutes a new MySQL BIN log file of 100MB. And every time a Drupal node_save() was called, "all" cache was just blown away ... I still don’t believe this is a very efficient way of flushing the cache. I tried a conversation on D.O.: http://drupal.org/node/115319#comment-5243634, but as you can read, changes would only effect Drupal 8.
• There was no replication system to flush the remote cache on the other web instance. Therefore, we had to write a custom mechanism. This mechanism has been explained in my other blogpost (I’m not gonna repeat it again). I still doubt this mechanism was working efficient ... It was working yes, but still ... Very amateurish.
• We were having issues with duplicate SQL entries on the semaphore table. The reason why is again explained in the previous blogpost. Solution can be found over there either. But this was still introducing us some other problem. Now Drupal could rebuild the table menu_router at both instances at the same time. The lock to this rebuilding mechanism is created by an entry in the semaphore table. But this lock wasn't replicated as we didn't replice the the semaphore table. menu_router is using the path column as a primary key. path is varchar. So again, we received duplicate SQL entries on this one. The only way to solve this problem was to manually skip the blocking SQL queries (which you don’t like doing when it happens at 3 O’Clock in the night :-) ...). A temporary solution was to whenever we wanted to rebuild the menu_router table (e.g.: you are flushing the menu cache, and believe me, this can happen a lot, even when you don’t know it), shutting down the second web instance. Wait for all traffic to be gone. If there's no traffic on the second web instance, the second web instance will not be rebuilding the menu_router table. Luckily, the load balancer would switch all traffic for us. Sessions are replicated too, so don’t worry about that one either. Then we could flush the menu cache. And when menu_router has been rebuilt, we could bring the second web instance again up.
• We were copying content files (which are located in /sites/##YOUR_SITE_NAME##/files) from server 1 to server 2 (and vice versa) using rsync. As rsync couldn't know if we were deleting a file, or if we were adding a file, we didn't allow deletes. Our htdocs folder was growing really fast. The monthly full backup was taking a lot of gigabytes.

The new architecture

So, after all these problems, we came up (also thanks to the hosting provider of my previous company) to this new setup which seems to me a better solution:

Memcache

• We moved the caching mechanism to memcache. To know more about Drupal and memcache, read the documentation on: http://drupal.org/project/memcache. Memcache can store all caching data in memory. So it doesn't use the MySQL tables any longer. Memcache is just way too awesome and bloody fast :-). Also, Memcache can run in a clustered environment, so no need to manually flush the remote cache. The memcache Drupal module and memcache daemon would take care of it
• Because of the movement of caching to memcache, our databases weren’t on heavy load any longer. Those MySQL cache queries were really hard to process for our MySQL daemons. The MySQL BIN log files are growing now very slow, which is perfect!

MySQL HA

Both MySQL daemons are clustered using MySQL High-Availability (MHA).

MHA performs automating master failover and slave promotion with minimal downtime, usually within 10-30 seconds. MHA prevents replication consistency problems and saves on expenses of having to acquire additional servers. All this with zero performance degradation, no complexity (easy-to-install) and requiring no change to existing deployments. MHA also provides scheduled online master switching, safely changing the currently running master to a new master, within mere seconds (0.5-2 seconds) of downtime (blocking writes only). MHA provides the following functionality, and can be useful in many deployments in which high availability, data integrity and near non-stop master maintenance are required.

So how does it work?

• MySQL HA serves 1 virtual database IP. Both web instances connect to that IP. In the background, 2 database instances are running in MASTER-MASTER replication, but one instance is active, the other is passive. The difference is made on a percentage. For example: database 01 is receiving percentage 150%, the other one 100%. But what does that mean? It means all queries coming from both the web instances are going to the MySQL daemon with the highest percentage. The other (passive) one is idle (no requests are coming in), but the passive one keeps track of all changes of the active one (using the well-known MySQL replication mechanism). If the active one goes out (e.g.: too much CPU load, database unavailable, ...), the percentage on the active one drops from 150% to 100%. The percentage of the passive one increases from 100% to 150%. So all requests will now be sent to the other database instance.
• This MySQL HA has one disadvantage: there’s a single point of failure. If the virtual DB IP is unreachable, all traffic to the web intances will be gone - they cannot contact the database any longer. Also, database 01 and database 02 must be strong enough to capture the MySQL CRUD queries from both web instances. But as long as you keep caching queries out of MySQL, I do think you’re fine.

File storage replication

A last improvement would be to store all data on a NAS file storage.

• The NAS storage holds all data in sites/##YOUR_SITE_NAME##/files directory. Compared with the previous solution, we don’t need to sync data again. And we can delete files, nice! I would suggest to have the source code on both web instances, and not on the file storage. This makes it possible to release new "source code" (not database!) instances of Drupal modules. Or you can quickly change some lines on a PROD environment for debugging (as long as you block traffic from visitors to that web instance of course ;-)).
• Again: one disadvantage here: if the NAS file storage goes out: no file in your files will be served. Nor by webinstance 1, nor by webinstance 2.

So, happy to have written this down :-). It took me a while! I hope you enjoyed reading this blogpost. Let me know what you think about this new architecture.

Spotted @ #416 Trance Around The World with Above & Beyond

Spotted @ DJ Tiësto's Club Life - podcast 256!

7 maand en 4 dagen heb ik er over gedaan om weg te geraken van de SMS dienst van Kokorico. Redelijk snel, niet? :-) *ironie*

Probleem

Het probleem dat ik had? Iedere vrijdag werd rond 17u een smsje uitgestuurd van Kokorico. Het smsje vertelde wat er dit weekend in de discotheek te doen was. Er werd niets aangerekend, SMS dienst was volledig gratis.
Toch wou ik op een bepaalde moment geen smsjes meer krijgen en ging ik op zoek naar een uitschrijfprocedure.

Oplossingen

• Uiteraard kijken we eerst naar de uitschrijfprocedure. Deze is terug te vinden op http://www.kokorico.be/info/#/faq_houserules en die leerde mij het volgende:
  

  To start or stop our weekly sms party info, please send an email to info@kokorico.be
  with the subject:
  START SMS 'YOUR NUMBER'
  or
  STOP SMS 'YOUR NUMBER'

  Dit heb ik 15 keer geprobeerd (met dan op de duur alle verschillende mogelijkheden in 'YOUR NUMBER' - zonder spaties, met spaties, scheidingstekens, ...). Maar uiteindelijk lost dit niets op. Waarschijnlijk zijn ze te tam om in te gaan op mijn vraag.

• STOP terugsturen naar het nummer dat je een smsje stuurde -> leverde niets op want het bericht kon nooit afgeleverd worden. Het smsje werd verstuurd van een echt GSM nummer (dus geen 4 cijferig nummer).
• Op de website van Kokorico staat een telefoonnummer. Maar op gelijk welk tijdstip ik belde, er pakte nooit iemand op.
• Kokorico aangesproken via Facebook en Twitter: geen reactie - op Facebook werd mijn bericht zelfs onmiddellijk verwijderd - zegt volgens mij genoeg?
• Ik zit momenteel bij Mobistar, dus heb ik bij Mobistar langs geweest om te vragen dit nummer te blokkeren - na heel wat telefoonwerk tussen de winkelbediende en Mobistar, kon dit blijkbaar niet.
• 3 keer naar de Mobistar helpdesk gebeld - deze mensen konden het nummer ook niet blokkeren (omdat de berichten gestuurd werden van een echt GSM nummer, en geen 4 cijferig nummer).

De échte oplossing

Wanneer je de nieuwsbrief van Kokorico krijgt, staat er onderaan de mail: Change email address / Leave mailing list. Wanneer ik op die link klikte, kwam ik op een pagina uit met mijn profielgegevens. Daar stond ... jawel ... mijn GSM nummer! Door daar mijn GSM nummer te veranderen (random phone generator? :p) slaagde ik er in van die str*nt SMS dienst af te geraken.

Dus Kokorico, bedankt om mij lang te spammen :-). Ik ben er uiteindelijk dan toch maar mooi van afgeraakt ;-)!