PHP 5.3 and friends on Karmic
I have been patient. For months I hoped that Ubuntu would sooner or later get PHP 5.3, a release I'm very much looking forward to, mainly because of the addition of anonymous inner functions to spell the death of create_function or even eval.
We didn't get 5.3 for Karmic and who knows about Lucid even (it's crazy that nearly one year after the release of 5.3, there is still debate on whether to include it in the next version of Ubuntu that will be the current LTS release for the next four years. This is IMHO quite the disservice against PHP 5.3 adoption).
Anyways: We are in the process of releasing a huge update to PopScan that is heavily focussed on getting rid of cruft, increasing speed all over the place and increasing overall code quality. Especially the last part could benefit from having 5.3 and seeing that at this point PopScan already runs well on 5.3, I really wanted to upgrade.
In comes Al-Ubuntu-be, a coworker of mine and his awesome Debian packaging skills: Where there are already a few PPAs out there that contain a 5.3 package, Albe went the extra step and added not only PHP 5.3 but quite many other packages we depend upon that might also be useful to my readers. Packages like APC, memcache, imagick and xdebug for development.
While we can make no guarantees that these packages will be maintained heavily, they will get some security update treatment (though highly likely by version bumping as opposed to backporting).
So. If you are on Karmic (and later Lucid if it won't get 5.3) and want to run PHP 5.3 with APC and Memcache, head over to Albe's PPA.
Also, I'd like to take the opportunity to thank Albe for his efforts: Having a PPA with real .deb packages as opposed to just my self-compiled mess I would have done gives us a much nicer way of updating existing installations to 5.3 and even a much nicer path back to the original packages once they come out. Thanks a lot.
Snow Leopard and PHP
Earlier versions of Mac OS X always had pretty outdated versions of PHP in their default installation, so what you usually did was to go to entropy.ch and fetch the packages provided there.
Now, after updating to Snow Leopard you'll notice that the entropy configuration has been removed and once you add it back in, you'll see Apache segfaulting and some missing symbol errors.
Entropy has not updated the packages to snow leopard yet, so you could have a look at PHP that came with stock snow leopard: This time it's even bleeding edge: Snow Leopard comes with PHP 5.3.0.
Unfortunately though, some vital extensions are missing, most notably for me, the PostgeSQL extension.
This time around though, Snow Leopard comes with a functioning PHP development toolset, so there's nothing stopping you to build it yourself, so here's how to get the official PostgreSQL extension working on Snow Leopard's stock php:
- Make sure that you have installed the current Xcode Tools. You'll need a working compiler for this.
- Make sure that you have installed PostgreSQL and know where it is on your machine. In my case, I've used the One-click installer from EnterpriseDB (which persisted the update to 10.6).
- Now that Snow Leopard uses a full 64bit userspace, we'll have to make sure that the PostgreSQL client library is available as a 64 bit binary - or even better, as an universal binary.Unfortunately, that's not the case with the one-click installer, so we'll have to fix that first:
- Download the sources of the PostgreSQL version you have installed from postgresql.org
- Open a terminal and use the following commands:
% tar xjf postgresql-[version].tar.bz2 % cd postgresql-[version] % CFLAGS="-arch i386 -arch x86_64" ./configure --prefix=/usr/local/mypostgres % make
make will fail sooner or later because you the postgres build scripts can't handle building an universal binary server, but the compile will progress enough for us to now build libpq. Let's do this:
% make -C src/interfaces % sudo make -C src/interfaces install % make -C src/include % sudo make -C src/include install % make -C src/bin % sudo make -C src/bin install
- Download the php 5.3.0 source code from their website. I used the bzipped version.
- Open your Terminal and cd to the location of the download. Then use the following commands:
% tar -xjf php-5.3.0.tar.bz2 % cd php-5.3.0/ext/pgsql % phpize % ./configure --with-pgsql=/usr/local/mypostgres % make -j8 # in case of one of these nice 8 core macs :p % sudo make install % cd /etc % cp php.ini-default php.ini
- Now edit your new php.ini and add the line
extension=pgsql.so
And that's it. Restart Apache (using apachectl or the System Preferences) and you'll have PostgreSQL support.
All in all this is a tedious process and it's the price us early adopters have to pay constantly.
If you want an honest recommendation on how to run PHP with PostgreSQL support on Snow Leopard, I'd say: Don't. Wait for the various 3rd party packages to get updated.
Tunnel munin nodes over HTTP
Last time I've talked about Munin, the one system monitoring tool I feel working well enough for me to actually bother to work with. Harsh words, I know, but the key to every solution is simplicity. And simple Munin is. Simple, but still powerful enough to do everything I would want it to do.
The one problem I had with it is that the querying of remote nodes works over a custom TCP port (4949) which doesn't work behind firewalls.
There are some SSH tunneling solutions around, but what do you do if even SSH is no option because the remote access method provided to you relies on some kind of VPN technology or access token.
Even if you could keep a long-running VPN connection, it's a very performance intensive solution as it requires resources on the VPN gateway. But this point is moot anyways because nearly all VPNs terminate long running connections. If re-establishing the connection requires physical interaction, then you are basically done here.
This is why I have created a neat little solution which tunnels the munin traffic over HTTP. It works with a local proxy server your munin monitoring process will connect to and a little CGI-script on the remote end.
This will cause multiple HTTP connections per query interval (the proxy uses Keep-Alive though so it's not TCP connections we are talking about - it's just hits in the access.log you'll have to filter out somehow) because it's impossible for a CGI script to keep the connection open and send data both ways - at least not if your server-side is running plain PHP which is the case in the setup I was designing this for.
Aynways - the solution works flawlessly and helps me to monitor a server behind one hell of a firewall and behind a reverse proxy.
You'll find the code here (on GitHub as usual) and some explanation on how to use it is here.
Licensed under the MIT license as usual.
Monitoring servers with munin
If you want to monitor runtime parameters of your machines, there are quite many tools available.
But in the past, I've never been quite happy with any of them. Some didn't work, others didn't work right and some others worked ok but then stopped working all of a sudden.
All of them were a pain to install and configure.
Then, a few days ago, I found Munin. Munin is optimized for simplicity, which makes it work very, very well. And the reports actually look nice and readable which is a nice additional benefit.
Apache parameters
Like many other system monitoring solutions, Munin relies on custom plugins to access the various system parameters. Unlike other solutions though, the plugins are very easy to write, understand and debug which encourages you to write your own.
Adding additional servers to be watched is a matter of configuring the node (as in "apt-get install munin-node") and adding two lines to your master server configuration file.
Adding another plugin for a new parameter to monitor is a matter of creating one symlink and restarting the node.
Manifestation of a misconfigured cronjob
On the first day after deployment the tool already proved useful in finding a misconfigured cronjob on on server which ran every minute for one hour every second hour instead of once per two hours.
Munin may not have all the features of the foll-blown solutions, but it has three real advantages over everything else I've seen so far:
- It's very easy to install and configure. What good is an elaboration solution if you can never get it to work correctly?
- It looks very nicely and clean. If looking at the reports hurts the eyes, you don't look at them or you don't understand what they want to tell you.
- Because the architecture is so straight-forward, you can create customized counters in minutes which in the end provides you with a much better overview over what is going on.
The one big drawback is that the master data collector needs to access the monitored servers on port 4949 which is not exactly firewall-friendly.
Next time, we'll learn how to work around that (and I don't mean the usual ssh tunnel solution).
Listen to your home music from the office
My MP3 collection is safely stored on shion, on a drobo mounted as /nas. Naturally, I want to listen to said music from the office - especially considering my fully routed VPN access between the office and my home infrastructure and the upstream which suffices for at least 10 concurrent 128bit streams (boy - technology has changed in the last few years - I remember the times where you couldn't reliably stream 128 bit streams - let alone my 160/320 mp3s).
I've tried many things so far to make this happen:
- serve the files with a tool like jinzora. This works, but I don't really like jinzora's web interface and I was never able to get it to work correctly on my Ubuntu box. I was able to trace it down to null bytes read from their tag parser, but the code is very convoluted and practically unreadable without putting quite some effort into that. Considering that I didn't much like the interface in the first place, I didn't want to invest time into that.
- Use a SlimServer (now Squeezecenter) with a softsqueeze player. Even though I don't use my squeezebox (an original model with the original slimdevices brand, not the newer Logitech one) any more because the integrated amplifier in the Sonos players works much better for my current setup. This solution worked quite ok, but the audio tends to stutter a bit at the beginning of tracks, indicating some buffering issues.
- Use iTune's integrated library sharing feature. This seemed both undoable and unpractical. Unpractical because it would force me to keep my main mac running all the time and undoable because iTunes sharing can't pass subnet boundaries. Aside of that, it's a wonderful solution as audio doesn't stutter, I already know the interface and access is very quick and convenient.
But then I found out how to make the iTunes thing both very much doable and practical.
The network boundary problem can be solved using Network Beacon, a ZeroConf proxy. Start the application, create a new beacon. Chose any service name, use «_daap._tcp.» as service type, set the port number to 3689, enable the host proxy, keep the host name clear and enter the IP address of the system running iTunes (or firefly - see below).
Oh, and the target iTunes refuses to serve out data to machines in different subnets, so to be able to directly access a remote iTunes, you'd also have to set up an SSH tunnel.
Using Network Beacon, ZeroConf quickly begins working across any subnet boundaries.
The next problem was about the fact that I was forced to keep my main workstation running at home. I fixed that with Firefly Media Server for which even a pretty recent prebuilt package exists for Ubuntu (apt-get install mt-daapd).
I've installed that, configured iptables to drop packets for port 3689 on the external interface, configured Firefly to use the music share (which basically is a current backup of the itunes library of my main workstation - rsync for the win).
Firefly in this case even detects the existing iTunes playlists (as the music share is just a backup copy of my iTunes library - including the iTunes Library.xml), though smart playists don't work, but can easily be recreated in the firefly web interface.
This means that I can access my complete home mp3 library from the office, stutter free, using an interface I'm well used to, without being forced to keep my main machine running all the time.
And it isn't even that much of a hack and thus easy to rebuild should the need arise.
I'd love to not be forced to do the Network Beacon thing, but avahi doesn't relay ZeroConf information across VPN interfaces.
Ubuntu 8.04
I'm sure that you have heard the news: Ubuntu 8.04 is out.
Congratulations to Canonical and their community for another fine release of a really nice Linux distribution.
What prompted me to write this entry though is the fact that I have updated shion from 7.10 to 8.04 this afternoon. Over a SSH connection.
The whole process took about 10 minutes (including the download time) and was completely flawless. Everything kept working as it was before. After the reboot (which also went flawlessly), even OpenVPN came back up and connected to the office so I could have a look at how the update went.
This is very, very impressive. Updates are tricky. Especially considering that it's not one application that's updated, not even one OS. It's a seemingly random collection of various applications with their interdependencies, making it virtually impossible to test each and every configuration.
This shows that with a good foundation, everything is possible - even when you don't have the opportunity to test for each and every case.
Congratulations agin, Ubuntu team!
git branch in ZSH prompt
Today, I came across a little trick on how to output the current git branch on your bash prompt. This is very useful, but not as much for me as I'm using ZSH. Of course, I wanted to adapt the method (and to use fewer backslashes 
).
Also, in my setup, I'm making use of ZSH's prompt themes feature of which I've chosen the theme "adam1". So let's use that as a starting point.
- First, create a copy of the prompt theme into a directory of your control where you intend to store private ZSH functions (~/zshfuncs in my case).
cp /usr/share/zsh/4.3.4/functions/prompt_adam1_setup ~/zshfuncs/prompt_pilif_setup
- Tweak the file. I've adapted the prompt from the original article, but I've managed to get rid of all the backslashes (to actually make the regex readable) and to place it nicely in the adam1 prompt framework.
- Advise ZSH about the new ZSH function directory (if you haven't already done so).
fpath=(~/zshfunc $fpath)
- Load your new prompt theme.
prompt pilif
And here's the adapted adam1 prompt theme:
# pilif prompt theme prompt_pilif_help () { cat <<'EOF' This prompt is color-scheme-able. You can invoke it thus: prompt pilif [<color1> [<color2> [<color3>]]] This is heavily based on adam1 which is distributed with ZSH. In fact, the only change from adam1 is support for displaying the current branch of your git repository (if you are in one) EOF } prompt_pilif_setup () { prompt_adam1_color1=${1:-'blue'} prompt_adam1_color2=${2:-'cyan'} prompt_adam1_color3=${3:-'green'} base_prompt="%{$bg_no_bold[$prompt_adam1_color1]%}%n@%m%{$reset_color%} " post_prompt="%{$reset_color%}" base_prompt_no_color=$(echo "$base_prompt" | perl -pe "s/%{.*?%}//g") post_prompt_no_color=$(echo "$post_prompt" | perl -pe "s/%{.*?%}//g") precmd () { prompt_pilif_precmd } preexec () { } } prompt_pilif_precmd () { setopt noxtrace localoptions local base_prompt_expanded_no_color base_prompt_etc local prompt_length space_left local git_branch git_branch=`git branch 2>/dev/null | grep -e '^*' | sed -E 's/^\* (.+)$/(\1) /'` base_prompt_expanded_no_color=$(print -P "$base_prompt_no_color") base_prompt_etc=$(print -P "$base_prompt%(4~|...|)%3~") prompt_length=${#base_prompt_etc} if [[ $prompt_length -lt 40 ]]; then path_prompt="%{$fg_bold[$prompt_adam1_color2]%}%(4~|...|)%3~%{$fg_bold[white]%}$git_branch" else space_left=$(( $COLUMNS - $#base_prompt_expanded_no_color - 2 )) path_prompt="%{$fg_bold[$prompt_adam1_color3]%}%${space_left}<...<%~ %{$reset_color%}$git_branch%{$fg_bold[$prompt_adam1_color3]%} $prompt_newline%{$fg_bold_white%}" fi PS1="$base_prompt$path_prompt %# $post_prompt" PS2="$base_prompt$path_prompt %_> $post_prompt" PS3="$base_prompt$path_prompt ?# $post_prompt" } prompt_pilif_setup "$@" |
The theme file can be downloaded here
Shell history stats
It seems to be cool nowadays to post the output of a certain unix command to ones blogs. So here I come:
pilif@celes ~
% fc -l 0 -1 |awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
467 svn
369 cd
271 mate
243 git
209 ssh
199 sudo
184 grep
158 scp
124 rm
115 ./clitest.sh
clitest.sh is a small little wrapper around wget which I use to do protocol level debugging of the PopScan Server.
Impressed by git
The company I'm working with is a Subversion shop. It has been for a long time - since fall of 2004 actually where I finally decided that the time for CVS is over and that I was going to move to subversion. As I was the only developer back then and as the whole infrastructure mainly consisted of CVS and ViewVC (cvsweb back then), this move was an easy one.
Now, we are a team of three developers, heavy trac users and truly dependant on Subversion which is - mainly due to the amount of infrastructure that we built around it - not going away anytime soon.
But none the less: We (mainly I) were feeling the shortcomings of subversion:
- Branching is not something you do easily. I tried working with branches before, but merging them really hurt, thus making it somewhat prohibitive to branch often.
- Sometimes, half-finished stuff ends up in the repository. This is unavoidable considering the option of having a bucket load of uncommitted changes in the working copy.
- Code review is difficult as actually trying out patches is a real pain to do due to the process of sending, applying and reverting patches being a manual kind of work.
- A pet-peeve of mine though is untested, experimental features developed out of sheer interest. Stuff like that lies in the working copy, waiting to be reviewed or even just having its real-life use discussed. Sooner or later, a needed change must go in and you have the two options of either sneaking in the change (bad), manually diffing out the change (hard to do sometimes) or just forget it and svn revert it (a real shame).
Ever since the Linux kernel first began using Bitkeeper to track development, I knew that there is no technical reason for these problems. I knew that a solution for all this existed and that I just wasn't ready to try it.
Last weekend, I finally had a look at the different distributed revision control systems out there. Due to the insane amount of infrastructure built around Subversion and not to scare off my team members, I wanted something that integrated into subversion, using that repository as the official place where official code ends up while still giving us the freedom to fix all the problems listed above.
I had a closer look at both Mercurial and git, though in the end, the nicely working SVN integration of git was what made me have a closer look at that.
Contrary to what everyone is saying, I have no problem with the interface of the tool - once you learn the terminology of stuff, it's quite easy to get used to the system. So far, I did a lot of testing with both live repositories and test repositories - everything working out very nicely. I've already seen the impressive branch merging abilities of git (to think that in subversion you actually have to a) find out at which revision a branch was created and to b) remember every patch you cherry-picked.... crazy) and I'm getting into the details more and more.
On our trac installation, I've written a tutorial on how we could use git in conjunction with the central Subversion server which allowed me to learn quite a lot about how git works and what it can do for us.
So for me it's git-all-the-way now and I'm already looking forward to being able to create many little branches containing many little experimental features.
If you have the time and you are interested in gaining many unexpected freedoms in matters of source code management, you too should have a look at git. Also consider that on the side of the subversion backend, no change is needed at all, meaning that even if you are forced to use subversion, you can privately use git to help you manage your work. Nobody would ever have to know.
Very, very nice.
Closed Source on Linux
One of the developers behind the Linux port of the new Flex Builder for Linux has a blog post about how building closed source software for linux is hard
Mostly, all the problems boil down to the fact that Linux distributors keep patching the upstream source to fit their needs which clearly is a problem rooted in the fact that open source software is, well, open sourced.
Don't get me wrong. I love the concepts behind free software and in fact, every piece of software I've written so far has been open source (aside of most of the code I'm doing for my eployer of course). I just don't see why every distribution feels the urgue to patch around upstream code, especially as this issue applies to both open- and closed source software projects.
And worse yet: Every distribution adds their own bits and pieces - sometimes doing the same stuff in different ways and thus making it impossible or at least very hard for a third party to create add-ons for a certain package.
What good is a plugin system if the interface works slightly different on each and every distribution?
And think of the time you waste learning configuration files over and over again. To make an example: Some time ago, SuSE delivered an apache server that was using a completely customized configuration file layout, thereby breaking every tutorial and documentation written out there because none of the directives where in the files they are supposed to be.
Other packages are deliberately broken up. Bind for example often comes in two flavors: The server and the client, even though officially, you just get one package. Additionally, every library package these days is broken up in the real library and the development headers. Sometimes the source of these packages may even get patched to support such breaking up.
This creates an incredible mess for all involved parties:
- The upstream developer gets blamed for bugs she didn't cause because they were introduced by the packager.
- Third party developers can't rely on their plugins or whatever pluggable components to work across distributions if they work upstream
- Distributions have to do the same work over and over again as new upstream versions are released, thus wasting time better used for other improvements.
- End users suffer from the general disability of reliably installing precompiled third-party binaries (mysql recommends the use of their binaries, so this even affects open sourced software) and from the inability to follow online-tutorials not written for the particular distribution that's in use.
This mess must come to an end.
Unfortunately, I don't know how.
You see: Not all patches created by distributions get merged upstream. Sometimes, political issues prevent a cool feature from being merged, sometimes clear bugs are not recognized as such upstream and sometimes upstream is dead - you get the idea.
Solution like FHS and LSB tried to standardize many aspects of how linux distributions should work in the hope of solving this problem. Bureaucracy and idiotic ideas (german link, I'm sorry) are causing quite the bunch of problems lately, making it hard to impossible to implement the standards. And often the standards don't specify the latest and greatest parts of current technology.
Personally, I'm hoping that we'll either end up with one big distribution defining the "state of the art", with the others being 100% compatible or with distributions switching to pure upstream releases with only their own tools custom-made.
What do you think? What has to change in your opinion?