IMO, there are much better ways of doing it. Specifically, generating config fragments for each virtual host.

I wrote such a system years ago, and have always thought “I should package it up for debian”. There’s a moderate amount of work involved in doing so, and probably an even larger amount of support work afterwards. Worst of all, I’d have to write – or con someone else into writing – documentation for it (it’s easy enough to use that you can be SHOWN how to use it in 10 minutes, but that isn’t good enough for a distributed package).

I guess the point of this blog post is to find out if there’s sufficient interest to make it worth the effort. Feel free to comment or email me about it.

If anyone wants to look at the code, i can tar it up easily enough, and even write some simple/minimalist instructions.

anyway, here’s (a slightly edited version of) what i posted to LUV-main:

Preamble

I looked into apache’s mod_vhost stuff years ago, but IMO it’s just too inflexible and limited to be of much use. It’s fine if ALL your vhosts are identical…but not if you need different settings on some or all of them. As soon as you need a custom setting for one vhost, you have to step outside mod_vhost_alias and roll-your-own anyway. which means you now have two different methods of configuring your vhosts.

Instead, I’ve used my own vhosting system for years (long before mod_vhost), on dozens of servers…the largest of which had over 500 vhosts on it – the biggest problem was the enormous number of file handles the apache processes needed for log files.

After tweaking the kernel file-max etc limits, it actually worked without problem but I always intended to rewrite that bit of it to use a log splitter daemon, perhaps vlogger or similar. Unfortunately, I a) left that job before I had time to implement it, and b) never did figure out a good way to handle the error logs because apache doesn’t prefix the vhost name to ErrorLog output like it does for CustomLog outout (best compromise was to just have one combined error log rather than one per vhost….most web programmers/website operators never look at the error logs anyway. Personally, I find it invaluable, most people don’t even seem to think of looking at it)

Synopsis of Operation

1. Creating a new vhost is a 3-step operation:

1. create a user (with adduser), optionally with a postgres or mysql account of the same name.
2. edit /etc/virtuals/virtual-hosts.conf, copy/paste/edit one line
3. run “make”

In step 2, hardly anything needs to be changed from one vhost to another – in the most common case, just the account’s login name, and the domain name. There are a number of different “flags” that can be set for each vhost in this file – ssl, cgi, mason, htdig, webcheck, aliases, group, samba, canonify (use mod_rewrite to redirect back to the site’s canonical name), and more.

The Makefile runs perl scripts to generate the apache config fragment (and other config fragments – e.g. for htdig and samba), add/remove any IP addresses if needed (usually not, with name-based vhosts), set up the home directory structure, copy in any sample files (e.g. html forms and images for htdig), and reloads apache. It also checks in any changes to svn (or RCS. easily modified to use other revision control systems).

On machines which are part of a load-balanced setup, it even ssh-es into other machines in the array, runs “svn update” and “make” to ensure that the vhost is operational on the other machines too.

Finally, it concatenates all the active config fragments into one large file because apache reloads a LOT faster if it doesn’t have to load hundreds of little config files.

2. Removing a vhost is as simple as editing virtual-hosts.conf and commenting out or deleting the line, then running make again.

virtual-hosts.conf

The virtual-hosts.conf file is extremely useful. It gives me a single authoritative source of all the relevant details about EVERY vhost running on a machine. These details can be used by other scripts (e.g. cron jobs to run webalizer, analog, htdig indexing, webcheck, etc; cgi scripts to display a list of vhosts on the machine with clickable links, and more).

This single config file makes the whole system easily extensible: write a new script to parse the config file, and loop through each entry, performing some relevant operation. e.g. run webalizer for every vhost, dumping the output under ~/public_html/LOGS/ (with optional .htaccess file to control access). Adding optional generation of samba shares for each vhost was quite easy.

In theory, it wouldn’t be hard to put a GUI or web front-end on to it. Oddly enough, i’ve never bothered. And I’m not inclined to bother now (but wouldn’t object if anyone else did)

Database? no, thanks

I’ve occasionally thought about putting this config data into a database, but the benefits just aren’t anywhere near good enough to outweigh vi and all the benefits of text file configuration (especially comments!)

Packaging?

And, of course, one of these days, i should package it all up for debian. I’ve been saying that for years, and just haven’t got around to it. It wouldn’t take a huge amount of work to do so, there are only a few small things i need to do to automate the installation, things that i just do by hand in a few minutes when installing on a new box.

Before packaging it, I also want to automate the adduser part of the process, as well as creation of postgres or mysql accounts. Most importantly, I ought to separate the code from the templates in the config fragment generator scripts, so that the local sysadmin can edit them without having to edit the scripts….OK, all up not a tiny job but not particularly big either….just big enough to be ignored on my TODO list for years.

I’d definitely have to change the config file format from the current one-line (colon separated fields) to using a .ini style file. I’ve wanted to do that for a while but have inertially-resisted it so far because the one-line format is so convenient…unfortunately, I’ve added so many flags and features over the years that it’s seriously ugly – probably scary to someone who doesn’t know it already (i.e. anyone outside of the set of “me + a handful of other people”). I estimate this would take about a days worth of hacking

I’d also take the opportunity to clean up some of the cruft :-)

Essential Design Criteria

BTW, one crucial design criteria for my system is that it should *NEVER* under any circumstances overwrite any hand-edited changes.

This is implemented in a number of ways. First, it takes an md5sum of each vhost’s config fragment when it generates it. If/when the script is run again, it only re-generates the config fragment if the md5sum hasn’t changed. If it has changed, it generates the updated fragment as “.new” for manual merging if required. Secondly, when generating conf fragments it looks for specially named files in a directory for extra stuff to add before the <VirtualHost> line, between the <VirtualHost>…</VirtualHost> lines, and/or after the </VirtualHost> line. This is to provide another location to put custom changes for each vhost while still retaining the ability to have improvements/new features to the script automatically applied to existing vhosts.

The reason for this is that I utterly loathe and despise allegedly “easy” configuration tools that blow away custom hand-edited changes. I hate them so much that this was THE major incentive for developing my own vhosting system.

In Use

I’ve been using my system for over 10 years now. It grew out of earlier bits and pieces, but the current incarnation has been working since 1999, with the occasional new feature or update (e.g. to add support for apache2 as well as apache1, and minor changes to adapt to changes in debian’s apache packaging).

There are also a handful of ISPs and several small businesses around Australia running it because they’ve hired me to set up their web servers.

Finally…

My vhosting system isn’t a system for people who are scared of or dislike command-lines or text-file configs. It’s a system for people who love them, but want an easy way to automate a dull, repetitive task. Such automation is easy for CLI tools, and next to impossible for web/GUI tools.

a

That’s actually the wrong question.

The right question is ‘Why is /var/cache/apt/archives even relevant on a machine with a full mirror of debian?’

/etc/apt/sources.list has always supported file:/ URLs. for example, sources.list on my local debian mirror looks like this:

deb file:/home/ftp/debian unstable main contrib non-free
deb file:/home/ftp/debian-multimedia/ unstable main

With a file URL, apt-get doesn’t need to download any packages, and thus doesn’t need to cache anything in /var/cache/apt/archives. No need to keep it empty, it just is empty.

In a related post, Andrew Pollock suggests adding DPkg::Post-Invoke { “apt-get clean”; }; to /etc/apt/apt.conf.

That would be fine if you ONLY ever ran “apt-get update ; apt-get dist-upgrade”.

It’s not fine if you run “apt-get -d” to download packages first, then install a few packages with “apt-get install package”, and then run “apt-get dist-upgrade”.

If you try that, then the DPKG::Post-Invoke rule will clean out /var/cache/apt/archives when you install the individual package(s). When you then run the dist-upgrade, apt-get will have to download all the packages again.

Better just to get into the habit of running apt-get clean or autoclean occasionally when you want to reclaim some disk space.

a