Welcome to MekTrix 4.0

Shimmie now powers a site which gets half a million page views per day, serving 200GB of images :3

Re: Lack of updates, see Dan Kim’s blog for a far better illustrated parallel to my thoughts…

LightTPD is win. Apache + mod_php + mod_python + mod_perl = 100MB per process, of which there are 10. Lighty does all the same stuff, in a single 30MB process \o/ Switching over has made travmap fast again, leaving a proxied apache behind to run the more complicated, less popular things~

19:57 >>> Mithrandir has joined #cs
19:57 [Monty] Thank goodness, Mithrandir is back!
21:52 [Shish] Monty: why do you never greet me? D:
21:52 [Monty] classics
21:53 [Shish] Monty: you only greet old timers? fair enough...
21:53 [Monty] apache/php job
21:53 [Shish] Monty: get it away from me!
21:53 [Monty] ffs
21:53 [Shish] Monty: no offence, your job offers are appreciated, I just
              react that way to php...
21:53 [Monty] nobody seems that size screw i hate cb302
21:54 [Shish] Monty: I suppose cb302 could be described as a screw, and
              I didn't like it either :S
21:54 [Monty] yay!
21:55 * Shish huggles monty for being better at the turing test than
              several people he knows...
21:55 [Monty] <3
21:55 [Shish] <3

In further automation, I tired of logging in and opening and resizing xterms, so taking bits from the mythtv wiki, I set up my desktop to automatically log me in and start all my usual programs. Of particular note is xterm’s “geometry” option, which allows you to set the position and size from the command line. Also useful is the “class” option, so that those of us with awesome window managers (enlightenment) can set them to remember position / stacking layer / border style / etc, and assign the remembered values to windows with set classes. For example, whenever I start a window with class “xterm_starter” (ie, by running “xterm -class xterm_starter”), enlightenment will notice it, shrink it, and move it to the corner of my screen, ready to start things from~

Part one of some notes on automation:

Having tired of keeping ~20 shell accounts up to date with my preferred settings, I made a script to do it. Snippets which other people may find useful are:

Only doing things if an app is installed:

is_installed() {
    if type -p $1 > /dev/null ; then
        return 0
    else
        return 1
    fi
}

if is_installed mplayer ; then
    ... do mplayer specific things ...
fi

Outputting large amounts of text from the script into the config file:

cat > ~/.vimrc <<EOD
syntax enable
set autoindent
set autowrite
set nocompatible    " Use Vim defaults instead of 100% vi compatibility
set backspace=indent,eol,start  " more powerful backspacing
set ruler
set foldmethod=marker
if v:version >= 700
    set spelllang=en_gb
    set spellfile=~/.vimspell.en_gb.add
    set spellfile=~/.vimspell.add
endif
EOD

cat > ~/.inputrc <<EOD
"e[A": history-search-backward
"e[B": history-search-forward

set completion-ignore-case On
#set editing-mode vi
set mark-symlinked-directories On
EOD

cat > ~/.bash_logout <<EOD
#!/bin/bash

if [ "$SHLVL" = 1 ]; then
    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi
EOD
chmod +x ~/.bash_logout

Now using those, I have all my important settings stored in one shell script. In order to push settings to another host, I can run “cat slek2.sh | ssh user@host.com”, which doesn’t even create any temporary files on the other side of the connection \o/

(Note: the post title, and the script, are called “slek”, which was originally a typo on “skel”, but then got made into “shish’s linux enhancement kit”. Though it’s also tested and working fine on solaris…)

I forget exactly when the project was started, but monitoring started a year ago. Thus I shall reminice:

Originally, a bunch of people in an IRC channel I hang out in decided to play Travian. As there were about 20 of us, keeping track was a pain, so I decided to make a map. This first version of the map was an SVG file drafted in Inkscape, edited by hand, and saved as a PNG image.

Pretty soon I got bored of updating by hand, and started looking for a way to automate things. Thankfully, the travian creators decided to be awesome and give out a dump of part of their database \o/ I put it into a local SQLite database so that it could be searched easily. Seeing as this was just a 5 minute hack that nobody other than myself would ever use or even see, I decided to use PHP to write the front end…

After a couple of weeks of adding features, I noticed that it could be useful to other people, so I posted about it on the forums. It ended up being quite popular, and I got several feature requests~ I also added support for other servers, and from there, to other languages. Having people offer to help with translation has been very motivational — It’s nice to know that people care about the project~

Talking about other languages, an interesting note on culture; when an American server is added, I get bug reports along the lines of “Type: Defect // Priority: Critical // The new server was added 3 hours ago, why is it not on travmap yet?!”. When a European server is added, the report is more like “Type: Enhancement // Priority: Minor // There is a new travian server at s2.travian.foo, please can you add it to your list? Thankyou :)”

Another few weeks later I had broken all my web serving records; I was serving tens of megabytes per day, and my server was melting — getting a P2 200MHz box with 64MB RAM to search through 200MB of database several times a second was painful. Switching to highly indexed MySQL made things better, but still bad. I ended up buying a load more RAM, ending up at 256MB. This was OK for a couple of months, until the database grew to be again too large to fit in memory… It got to the point where it was being queried faster than it could respond, so it got slower, and things stacked up exponentially — load average broke 60 within minutes of apache being started :-/

Eventually I gave up on self hosting, and moved across to dreamhost — thanks to the wonders of overselling, they had tons of spare processing power, memory, and bandwidth. Aside from datacenter problems, they’ve been pretty cool.

Since then, growth has been pretty steady. The database is now over 500MB, going over a hit a second at peak times (That’s quite a lot of traffic, nearing a gig a day now…). I also notice that dreamhost moved me to a different database server without telling me :|

Also I figured that with 10,000 visitors per day, there ought to be some way of making some money out of it — putting up a “please pay for my hosting” link earned me a grand total of $5 ($4.50 once paypal took their cut) from one guy — to this day, the only money I’ve made directly from programming ^_^ About a month ago I put some ads on the site to see how effective they were, and since then I’ve made enough to cover nearly half of that 256MB RAM I bought so long ago — another 3-4 months of the same earnings, and I’ll have got as much money out of the project as I’d put in XD

Overall, it’s been a fun year — I’ve learned lots about caching and optimisation, and how much PHP sucks (I *really* wish I’d written it in something else, I didn’t know it’d get so big…); I haven’t actually played travian for a few months now, I’m just sticking with the project because it’s enjoyable~

I forget how I was going to end my rambling, so I shall do so abruptly. End.

1) Akismet — I’ve been using it to stop wordpress and trac spam for the past ~3 months, and I only just remembered that it exists, since it let one through (as opposed to the 20 a day I was getting before)

2) OpenID; which has been around a while, but might actually get somewhere now that microsoft have said they’ll be putting their weight behind it (although they may well end up doing their embrace / extend / extinguish thing…)

I shall add support for them to my webapps once I find APIs and time…

So long since the last post~

I’m now half way through my summer job, and have been meeting a variety of interesting programs:

• Cacti, which I spent most of today with — it’s like MRTG + RRDTool, but with considerably less hand typing of huge config files.

• A variety of Cisco and HP routers, which I’ve been setting up monitoring for.
• Motion, for monitoring webcams, and recording whenever there’s activity in the room.

I have 20 autoconf / compiler processes running, my music plays without a jitter, and my UI is as responsive as when it has the whole CPU to itself — I never realised how apparent the new linux scheduler was until I saw that it was giving a single interactive process more time than all the batch processes combined~

Compare windows, where a single cpu / memory hogging program can render the whole computer unusable…

ED: Locking comments, since there are a couple of spams per day and no useful comments…

After 8 hours of being under full load, chai (my linux router) panicked, taking the network offline for 3 hours. There’s nothing *obviosuly* wrong with the hardware, so I assume it a linux problem. I would have investigated properly, but the stack trace only showed the addresses of functions, not names…

Seeing as one error per 4 years is unacceptably high (from debian stable, of all distros!), I’ve since started looking into the BSDs; particularly m0n0wall.