Downloading and extracting large tarballs usually takes plenty of time. Extracting tarballs during the their transfer reduces the waiting time because you a) don’t need to do wait for the transfer to finishes before starting the extraction and b) the machine doesn’t have to read the tarball from the disk while simultaneously writing to it.

One way to extract tarballs on the fly while downloading them is using wget to fetch the file and piping it’s output (the downloaded data) to tar.
wget --user="username" --password="my secret password" -O - "ftp://foo.bar/foobar.tgz" | tar -xvzf -

Hint: You may want to turn hoff Bash’s history expansion if your username or password contains exclamation marks. The fastest way doing so is starting a new bash process with the corresponding argument (+H):
bash +H
wget...

MSDOS MBR Partition Tables make it impossible to use Partitions larger than 2 TiB. This is a pitfall you maybe will be faced with after expension of a RAID array. The only solution is to convert the MBR Partition Table to GPT (GUID Partition Table).

This can be done quite easily by using GPT fdisk.
GPT fdisk includes the sgdisk program which can convert MBR Partition Tables to GUID Partition Tables using the following command:

sgdisk -g device

(for example sgdisk -g /dev/sda)

Warning: doing this will make your system unbootable as there’s no longer a master boot record existing.
The BIOS Boot Partition replaces the MBR. Make sure you have enough unused space to create it.

If you start Solr with the default configuration, there will be only one instance running. That’s enough if you have only one website which you want to index but if you have several websites, you probably don’t want them to share an index as you then would get search results from both websites.
Luckily Solr already brings the so called multicore feature with it. It allows you to run multiple instances that use different configuration sets and indexes.

This little howto will show you how to install / configure Solr to make use of the multicore feature.

1. Install the Sun Java Runtime Environment
On Debian Linux you will do this by executing apt-get install sun-java6-jre

2. Choose the directory in which you want to install Solr. In this example we will use /opt.
If it doesn’t exist already, create the directory and cd to it.
cd /opt

3. Go to http://www.apache.org/dyn/closer.cgi/lucene/solr/ and choose a mirror. In this example we will use http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist//lucene/solr/. Download the latest Solr version, currently 1.4.1:
wget http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist//lucene/solr/1.4.1/apache-solr-1.4.1.tgz

4. Extract the downloaded archive:
tar xzf apache-solr-1.4.1.tgz

5. Copy the Solr examples directory. In this example we will copy it to /opt/solr/.
cp -R apache-solr-1.4.1/example/ solr

6. cd to /opt/solr/
cd /opt/solr/

7. Remove the following directories (you probably won’t need them):
- example-DIH
- exampledocs
- work

rm -Rf example-DIH exampledocs work

8. Copy /opt/solr/multicore/solr.xml to /opt/solr/solr/solr.xml and remove the multicore directory.
solr.xml describes the name and location of the cores to use.
cp /opt/solr/multicore/solr.xml /opt/solr/solr/ && rm -Rf /opt/solr/multicore/

9. Edit /opt/solr/solr/solr.xml and modify it to match your needs. For example:
<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="true">
<cores adminPath="/admin/cores">
<core name="myfirstwebsite" instanceDir="site_myfirstwebsite" />
<core name="mysecondwebsite" instanceDir="site_mysecondwebsite" />
</cores>
</solr>

10. Create the directories you specified as "instanceDir" in step 9. Copy /opt/solr/solr/conf into them and create data directories.
mkdir -p /opt/solr/solr/site_myfirstwebsite/data && cp -R /opt/solr/solr/conf/ /opt/solr/solr/site_myfirstwebsite/
mkdir -p /opt/solr/solr/site_mysecondwebsite/data && cp -R /opt/solr/solr/conf/ /opt/solr/solr/site_mysecondwebsite/

11. Edit /opt/solr/solr/site_myfirstwebsite/conf/solrconfig.xml and /opt/solr/solr/site_mysecondwebsite/conf/solrconfig.xml. Change the dataDir-tag to match the new data directories you created in step 10:
<dataDir>/opt/solr/solr/site_myfirstwebsite/data</dataDir> respectively <dataDir>/opt/solr/solr/site_mysecondwebsite/data</dataDir>

12. Start Solr. We will use Jetty here because we wouldn’t need Tomcat for anything else but Solr.
java -jar -server -Xmx128M -Dfile.encoding=UTF8 -Djava.headless=True /opt/solr/start.jar

After Solr has been startet you can access the Solr Admin at http://SOLR_HOSTNAME:8983/solr/ (you should find a link pointing to the Solr Admin for every core you have configured before). Solr queries need to go to http://SOLR_HOSTNAME:8983/solr/CORE_NAME/, f.exp. http://192.168.10.41:8983/solr/myfirstwebsite/select/?q=*%3A*&version=2.2&start=0&rows=10&indent=on

If you use a combination of Redmine.pm, Apache and SVN and are wondering about error messages telling you "COPY of ‘foobar’ : Could not read status line: connection was closed by server" or "COPY of ‘foobar’ : Could not read status line: secure connection truncated" you probably will want to switch to apache2-mpm-prefork to get rid of them.

A combination of Redmine.pm, apache2-mpm-worker and Redmine.pm doesn’t seem to be stable at all.

How to create encrypted loopback images with dm-crypt and LUKS + automatically mounting them after login with pam_mount

I recommend using debian squeeze for this scenario as lenny includes a very old version of libpam-mount and I had lots of problems when I tried using it.
Using only the libpam-mount package and its dependencies from squeeze maybe (I didn’t try it and I wouldn’t recommend it either) does the job too, but at least has a very bitter after taste if you take a closer look at the dependencies.

1. Make sure you have the required kernel modules loaded. If you use the stock debian kernel, this will be the case. if you don’t, make sure you’ve set the following options:

• CONFIG_BLK_DEV_DM=y or CONFIG_BLK_DEV_DM=M
• CONFIG_DM_CRYPT=y or CONFIG_DM_CRYPT=M
• CONFIG_CRYPTO_CBC=y

Additionally, you need to include support for at least one cipher.

In make menuconfig, you can find the required kernel modules at the following locations:

Device Drivers  --->
	Multi-device support (RAID and LVM)  --->
		<*> Device mapper support
		<*> Crypt target support

Cryptographic options  --->
	 SHA256 digest algorithm
	 AES cipher algorithms (x86_64)

To avoid a reboot, you can build all of these options as modules. If you chose to do so, you can later load the modules by using modprobe .

2. Install the required packages
apt-get install cryptsetup libpam-mount
…apt-get should take care of all dependencies

3. Generate a random key and assign it to a variable for later use

KEY=`tr -cd [:graph:] < /dev/urandom | head -c 79`

4. Encrypt the key and save it to a file

echo $KEY | openssl aes-256-cbc > container.key

5. Create the loopback file and fill it with random data

dd if=/dev/urandom of=~/container.img bs=1G count=10

This will create a 10GB file and fill it with random data taken from /dev/urandom.
Another option (which will be much faster especially on older hardware) is using /dev/zero to fill the loopback file with zeros:

dd if=/dev/zero of=~/container.img bs=1G count=10

6. Set up a loop device

losetup /dev/loop0 ~/container.img

7. LuksFormat it

echo $KEY | cryptsetup -v -c aes -s 256 luksFormat /dev/loop0

8. Open it

cryptsetup luksOpen /dev/loop0 container

9. Make a filesystem of your choice

mkfs.xfs /dev/mapper/container

10. Close it and delete loop

cryptsetup luksClose container && losetup -d /dev/loop0

11. Configure pam_mount
Open /etc/security/pam_mount.conf.xml in your favorite text editor and change it to the following:

<?xml version="1.0" encoding="utf-8" ?>

<!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd">
<pam_mount>
	<debug enable="1" />
	<mkmountpoint enable="1" remove="true" />

	<msg-sessionpw>reenter password for pam_mount:</msg-sessionpw>
	<volume user="foobar" path="/home/foobar/container.img" mountpoint="/home/foobar/containercontents"
		options="cipher=aes-cbc-essiv:sha256,hash=sha512,keysize=256" fstype="crypt" fskeycipher="aes-256-cbc"

		fskeypath="/home/foobar/container.key" fskeyhash="md5" />
</pam_mount>

Using this configuration the image /home/foobar/container.img will get mounted into /home/foobar/containercontents when the user foobar logs in.
Enabling debugging is pretty usefull if something isn't working as it should. In this case you can take a look at /var/log/auth.log.

12. Include /etc/pam.d/common-pammount in the PAM configuration files of the services that should use it (for example: SSHd)
Open /etc/security/sshd in your favorite text editor. Look for the line "@include common-session" and add a new line after it:

...
@include common-session
@include common-pammount
...

13. If needed, change the configuration of the relevant services (for example: SSHd)
Open /etc/ssh/sshd_config in your favority text editor and make sure you have the following lines in there:

# pam_mount
UsePAM yes
PasswordAuthentication yes
ChallengeResponseAuthentication no
UsePrivilegeSeparation no
PermitUserEnvironment yes

If you disable PasswordAuthentication and use keys instead you have to enter the users password after connecting via SSH.

14. Test if anything works as expected
Open a root session or use sudo and watch the auth log by using tail -f /var/log/auth.log. Then login as the user for which you have configured a volume earlier.
If the encrypted loopback image gets mounted, also test if it gets unmounted again, when the user logs out.
If anything works remove the debug line from /etc/security/pam_mount.conf.xml.

Many thanks go to the users tuxophil and pillgrim from the gentoo forums. Large parts of this howto were taken from their postings at http://forums.gentoo.org/viewtopic-t-274651.html.

When I was trying to automatically mount an encrypted image at login using pam_mount, I encountered a strange problem:
I wasn’t able to find any errors in my configuration (at least none connected to this behaviour), but mount.crypt was unable to mount the image.
Enabling debugging in pam_mount.conf.xml (<debug enable="1" />) revealed the command used for mounting:

pam_mount(misc.c:272) command: mount.crypt [-o loop] [/home/foobar/container] [/home/foobar/crypto]

In Debians libpam-mount 0.44-1+lenny3, /sbin/mount.crypt is a bash script that calls cryptsetup with parameters generated during its runtime.
Debugging it (set -x), I stumpled upon the following command:

cryptsetup -c aes -h ripemd160 -s 256 create _home_foobar_container /home/foobar/container

…which returned the following error message:

Command failed: BLKROGET failed on device: Inappropriate ioctl for device

So mount.crypt has been passed the loop option but cryptsetup was told to mount the encrypted image instead of the associated loopback device (normally mount.crypt executes losetup to create this association in its function _losetup, but this apparently has never been called).

Long story short:
There seems to be a bug in Debians libpam-mount 0.44-1+lenny3.
The case statement at line 110 in /sbin/mount.crypt doesn’t work as expected because the variable $KEY used for option comparison has an ordinary space at its values beginning.
To fix this, you can either change the way how passed options are saved in the OPTION variable (starting at line 60) or change line 110 to the following:

case ${KEY/ /} in

This will remove all spaces.

Eigentlich ist suPHP eine nette Sache. Das Apache Modul mod_suphp sorgt in Kombination mit einem SetUID Binary dafür, dass PHP-Skripte mit den Berechtigungen des Benutzers, dem sie gehören, ausgeführt werden. Damit das funktioniert muss man in Binär-Distributionen wie etwa Debian Linux nicht mehr tun, als dass Modul zu installieren, einen Blick auf dessen Konfiguration zu werfen und es anschließend zu aktivieren. Vielleicht ist es gerade diese Einfachheit (deren Preis die Starrheit ist), die es so beliebt macht.

Ein großes Problem tut sich aber dann auf, wenn man PHP-Caches wie APC, TurckMM oder eAccelerator verwenden möchte. Denn all diese Caches funktionieren nicht im Zusammenspiel mit suPHP. Zwar lässt sich eine solche Kombination verwenden, doch ist sie leider nutzlos, da nichts gecached wird.

Eine wundervolle Alternative zu der Verwendung von suPHP ist eine Kombination aus Apache, SuExec, FCGID und PHP-CGI. Mit dieser Kombination funktioniert auch das Caching problemlos. Eine Anleitung zum Aufsetzen dieser Kombination findet sich bei Hetzner.de.

Wer seine Websites nicht in dem von der jeweils genutzten Linux Distribution dafür vorgesehenen Verzeichnis ablegt sondern stattdessen beispielsweise in /home/*/public_html/ muss darauf achten, SuExec selbst zu kompilieren oder auf Debian-Systemen die gepatchte Version apache2-suexec-common zu verwenden. Aus Sicherheitsgründen erwartet SuExec bereits bei der Kompilierung den Pfad zum Docroot. Bei der Verwendung von apache2-suexec-common lässt sich das Docroot in der Datei /etc/apache2/suexec/www-data angeben.

Das Loop Disk Image einer XEN DomU ist zu klein geworden? Solang die Festplatte des Hypervisors noch genügend Platzreserven hat, ist das glücklicherweise kein unlösbares Problem. Loop Disk Images können mit Bordmitteln vergrößert werden. Zwei kleine Beispiele für die Vorgehensweise bei EXT2, EXT3 und XFS Dateisystem:

EXT2 / EXT3 Dateisystem
VM herunterfahren:

xm shutdown domu.blub.local

Image um 30GB vergrößern:

dd if=/dev/zero bs=1024k count=30720 >> disk.img

Fehler im Dateisystem korrigieren:

e2fsck -f disk.img

Dateisystem auf die Größe des Images vergrößern:

resize2fs disk.img

XFS Dateisystem
VM herunterfahren:

xm shutdown domu.blub.local

Image um 30GB vergrößern:

dd if=/dev/zero bs=1024k count=30720 >> disk.img

Fehler im Dateisystem korrigieren:

xfs_repair -f disk.img

Dateisystem auf die Größe des Images vergrößern:

mount disk.img /mnt/tmp/ -o loop
xfs_growfs -d /mnt/tmp/

Mir sind gerade fast die Augen rausgefallen als ich bei einem Blick ins Error-Log des Apache Webservers unter Debian Etch abertausende Zeilen mit der Fehlermeldung "PHP Warning: Module ‘json’ already loaded in Unknown on line 0" sah. Jeder Seitenaufruf erzeugte diesen Fehler. Die installierten Pakete (php5-cgi + Module sowie apache2 + Module) entstammen dem offiziellen Debian Etch Repository, das Problem betrifft also wahrscheinlich auch noch viele weitere Systeme, bei denen nicht ständig ins Error-Log geschaut wird. Wahrscheinlich wird beim Start des php-cgi Prozesses noch eine andere Konfigurationsdatei als /etc/php5/cgi/php.ini, mit der Anweisung das Modul zu laden, ausgelesen.

Beheben lässt sich das Problem durch das Editieren von /etc/php5/cgi/php.ini. Dort die Zeile, die das json-Modul lädt, einfach auskommentieren und den Apache neustarten.

Was ist Code ohne Dokumentation? Genau: total daneben.

Zum Glück gibt es ja praktische Helfer wie phpDocumentor, um aus Kommentaren eine richtig anständige Dokumentation zu zaubern. Aber das verdammte Ding nach jedem SVN commit anzustoßen ist auf Dauer ganz schön nervig…

Abhilfe verspricht der post-commit hook. In dem Wurzelverzeichnis jedes SVN Repositorys befindet sich ein Verzeichnis namens hooks. Dort können beliebige ausführbare Dateien hinterlegt werden, die je nach deren Dateinamen zu einem bestimmten Zeitpunkt ausgeführt werden (Beispiele / Templates sind bereits in dem Verzeichnis enthalten).
Um phpDocumentor nach einem erfolgreichen SVN commit zu starten kann man ein Bash-script verwenden, indem man diesem den Dateinamen post-commit verpasst und das Executebit setzt (chmod +x foo).

Der Ablauf im Script ist dann wie folgt:
1. SVN checkout anwerfen
2. phpdoc -d anwerfen
3. Quellcode löschen (bei Bedarf)

Das Script bekommt bei seinem Aufruf 2 Argumente: den absoluten Pfad zum betroffenen Repository sowie die Reversion nach dem commit.

Besonders praktisch finde ich daran die Möglichkeit den Quelltext zusammen mit der entsprechenden Dokumentation im Intranet zugänglich zu machen.