pruperting

Stop snort detecting local traffic

I use the rather excellent and Open Source snort to help monitor and protect my network. However, it was causing muchos havoc with my VNC, SSH and samba connections. I found out this was all down to (as is sadly normal for most opensource projects) confusion related to the configuration.

In the snort.conf file, you are meant to set you home network (the place you are protecting, but don’t want to detect), using

HOME_NET XXX.XXX.X.XXX

And then your external network (the place where attacks might come from and do want to detect) using

EXTERNAL_NET XXX.XXX.X.XXX

Now, the method the configuration and snort documentation tells you to use is, to basically tell snort that anything that isn’t in your home network is in your external nework, so

EXTERNAL_NET !HOME_NETWORK

The exclamation basically meaning NOT. However, if you put an IP address string as your home network, for example (as is suggested in the configuration)

HOME_NET 192.168.1.1/24

for some reason, using !HOME_NETWORK for your external network doesn’t work and snort will happily go and detect all traffic coming from your local network…GRR. Luckily, after some Googling I came across this message and found that the way to get EXTERNAL_NETWORK to work correctly was to use

EXTERNAL_NET [!192.168.1./24]

Finally, snort no longer goes ape about my local traffic and now only finds the retarded script kiddies from Russia and China and the Netherlands (since that is where most of the attacks I get come from) and blocks them. Thanks mailing list, no thanks snort.conf….

Tags: hackers, security, snort

How To Use FFMPEG To Transcode Video For The PS3

So, it seems my last post actually wasn’t the best solution. Although using Mediatomb and VLC worked some of the time, often I found that the audio and video was out of sync, making it impossible to watch the movie.

So, I looked for another alternative. I already use HandBrakeCLI to transcode my videos, but that is essentially a front end to ffmpeg and x264. They are moving away from providing decent support for transcoding videos on the Playstation 3, so I decided to use ffmpeg directly. I used this guide to install the latest svn versions of ffmpeg and x264 on my Ubuntu 8.04 server. I then spent about two weeks trying to find an up-to-date command to actually use ffmpeg to create a PS3 compatible file. It seems most of the instructions on the web are way out of date and use commands that ffmpeg no longer accepts. Finally, after lots of trying I finally asked on the IRC channel #ffmpeg. This very nice dude called relaxed suggested the following:

ffmpeg -t 120 -i $input -vcodec libx264 -level 41 -vpre hq -crf 24 -threads 0 -acodec libfaac -ab 128k -ac 2 output.mp4

Sadly I had problems with this code, since the .mpg files I was using (I am now using MythTV instead of MediaPortal, but that’s another post) were not accepted by the libfaac codec.

However, with a little modification, I got the perfect command. So, without further ado, the best command I have found to convert ANY video accepted by ffmpeg into a PS3 compatible file using x264 that is good quality and low file size is:

ffmpeg -y -i "$1" -vcodec libx264 -level 41 -vpre normal -crf 24 -threads 0 -acodec libfaac -ab 128k -ac 2 -ar 48000 "$1".mp4

Obviously change $1 into your input file and “$1″.mp4 into your output file. If you want to send the command line output to a log file so you can see what is going in, use:

ffmpeg -y -i "$1" -vcodec libx264 -level 41 -vpre normal -crf 24 -threads 0 -acodec libfaac -ab 128k -ac 2 -ar 48000 "$1".mp4 > /where/you/store/your/logs/ffmpegprog.log 2>&1

Now, when I first saw this I thought, hey, its got hardly any switches (all the -XYZ stuff), therefore it must be cack.  However, the key is the “-vpre normal” switch.  This tells ffmpeg to use some of its built-in presets, or in this case built-in x264 presets that are created when you build x264 from SVN. I was initially told, as you can see from above, to use the “hq” preset, but I found this resulted in files that were quite big. Using the “normal” preset seemed to produce files that were of great quality compared to my original source (SD DVB-T transport streams in MPG format) but much smaller files size, for example, a .mpg transport stream movie that is originally 2.5GBs in size gets reduced to around 650 MBs, awesome.

So, this is the command I now use to convert all my videos. I also use comskip run via Wine to check for commercials and mp4box to mux in the chapter markers from comskip. Weirdly, I do find that when playing these resultant files via VLC they appear to have their audio and video out of sync, but on the PS3 they are fine. Whether this is due to me using an old VLC or my PC being too slow to play em, I don’t know, but hey, if it works.

So, if you are looking for the best ffmpeg command to transcode video files for the PS3 using ffmpeg and x264, I am pretty sure this is it.

As an aside, I also built HandBrakeCLI from SVN and it now also has a basic preset, called Normal, that also produces PS3 compatible files. I will play with them to see how they do, though I find that occasionally HandBrake can crash, and get stuck, so I try to avoid it. Sadly though, the HandBrake dev’s wont help you with this, since you are using the HandBrakeCLI and it is not the CLI that is crashing so they wont look into it for you. In fact (<rant>) I am really shocked at the rudeness and arrogance that the HandBrake devs demonstrate on their forum. I have read so many well worded and polite posts from confused people on their forums, where the devs (mainly a guy called jbrake) just answers back incredibly sarcastically and very rudely, providing little to no help. Maybe they get loads of annoying whiny posts from people who haven’t RTFM, but still, there is no need to be rude about it. In fact, that is the main reason I don’t want to use HandBrake, since I don’t like the attitude those guys have (</rant>).

Anyway, hope this awesome command for a recent SVN ffmpeg and x264 version to convert video files for PS3 works for you!

UPnP, PS3, Transcoding and MediaTomb

For quite a while I had been using PS3MediaServer to provide access to my Video files from my PS3. The problem was, I was getting huge problems getting it to support anything other than the files that the PS3 natively supports. I had previously messed around with MediaTomb but I couldn’t get it’s transcoding features to work. But then I came across this post and it sorted all my problems out. I simply followed the tutorial there and made some minor adjustments, which I will describe below.

MediaTomb

The quickest way to follow the tutorial is to cheat and just copy and paste the config.xml that is kindly provided. However, I have found that copy and pasting the whole config.xml doesn’t work. I am pretty sure it is to do with the <udn>uuid= part of the config, since I guess this aligns and individual config to an individual MediaTomb installation. Thus, simply copy and paste all the stuff after and including this line:

“<protocolInfo extend=”yes”/><!– For PS3 support change to “yes” –>

and you’ll be good to go. Next, since most of my files are in the MKV container, I needed to add some extra stuff to support MKV files and the transcoding of them. So, I added the following two lines to their respective places (you can tell where they should go by looking at their format and looking for other lines that look similar). I added these two lines:

<map from=”mkv” to=”video/x-matroska”/>

<transcode mimetype=”video/x-matroska” using=”vlcmpeg”/>

And awesomely, it worked straight out of the box. Nice one Mr Pyke! There are a few more things to add, like adding thumbnail support, which I understand you can do by using a script and app called thumbnailermpeg, but that’s about it. I have found that using MediaTomb and VLC to transcode files puts much less strain on my server than using PS3MediaServer ever did. HTOP shows MediaTomb and VLC when streaming an MKV file pushing the server to around 15% CPU usage on my dual core Pentium4. Doing the same thing for one of the few supported movies by PS3MediaServer shows CPU usage maxed out at 100% on both cores full time. Ouchy! So I have saved some energy and prolonged the lifespan of my server by doing this too. I know that at the moment MediaTomb doesn’t do all the fancy things that PS3MediaServer does (like AC3 pass-through and stuff) but I can live without that. I am sure there was a lot I was missing in PS3MediaServer that it can do and it is updated much more frequently than MediaTomb, but I just couldn’t agree with the huge resource usage it entails.

I now have full streaming of all my recorded-off-the-tele MKV movies and I am a happy bunny, so thanks RobertPyke and thanks MediaTomb!

MediaPortal Comskip and Transcode Bat and Bash Scripts

As I mentioned earlier, I use MediaPortal to record DVB-T broadcasts. As these end up quite big, I then use two scripts to check the recordings for adverts and to transcode the recordings to a mkv file (smaller and better supported).

This solution isn’t very elegant, since it is for my personal setup which is a little unusual, in that the recordings are done via a Windows XP MediaCentre PC, they are then analysed for adverts using Comskip and are then transferred to my Ubuntu-based server, which then transcodes the files to an MKV file and the adds Chapter markers to those MKV files using the file from Comskip. Thus there script is in two parts, one a Windows Bat script and the other is a Ubuntu Bash script.

The scripts are below. First, the Windows Bat script that runs the recorded file through Comskip and then copies it over to the server. Since I wanted to only run one script for all the recordings made in an evening, the script takes the file name from MediaPortal, which passes it to the script after the file is finished recording, using the Comskip plugin. This explains all the weird “Echo” in front of all the terms, since basically it is putting all the commands into another script, which is then run early each morning.

BASH

@ECHO OFF
ECHO ECHO.
ECHO Rem Batch commands for %1 START >> "C:\scripts\comskip\trans.bat"
ECHO CD "C:\scripts\comskip" >> "C:\scripts\comskip\trans.bat"
ECHO START "" "C:\Program Files\VideoLAN\VLC\vlc.exe" >> "C:\scripts\comskip\trans.bat"
ECHO %%SystemRoot%%\explorer.exe "Z:\" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO ************************** ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Post Processing started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO Starting mpeg2repair %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO mpeg2repairHelper.exe %1 %1-fixed.ts >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO mpeg2repair finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Comskip started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO comskip %1 >>"%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Comskip finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO %%date%% %%time%% ^>^> "Z:\new_files.log" >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO ComClean started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO mencoder "%~dpn1.ts" -edl "%~dpn1.edl" -alang eng,nar -noskip -vf harddup -oac copy -ovc copy -of mpeg -o "%~dpn1_clean.ts" >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO ComClean finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO New file %1 being moved to Server ^>^> "Z:\new_files.log" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Tidying started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO %%SystemRoot%%\explorer.exe "Z:\" >> "C:\scripts\comskip\trans.bat"
ECHO CD "%~dp1" >> "C:\scripts\comskip\trans.bat"
REM ECHO ren *.edl *.ts.edl >> "C:\scripts\comskip\trans.bat"
REM ECHO ren *.txt *.ts.txt >> "C:\scripts\comskip\trans.bat"
ECHO ren *.chap *.ts.chap >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Moving Clean TS to Server directory on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO IF EXIST "%~dpn1_clean.ts" del %1 >> "C:\scripts\comskip\trans.bat"
ECHO xcopy %1 Z:\ >> "C:\scripts\comskip\trans.bat"
ECHO IF EXIST "Z:\%~n1.ts" del %1 >> "C:\scripts\comskip\trans.bat"
REM ECHO xcopy "%~dpn1.edl" Z:\ >> "C:\scripts\comskip\trans.bat"
ECHO del "%~dpn1.txt" >> "C:\scripts\comskip\trans.bat"
ECHO xcopy "%~dpn1.ts.chap" Z:\ >> "C:\scripts\comskip\trans.bat"
ECHO ECHO TS and others moved to Server directory on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO %~n1.ts moved to Server on %%date%% %%time%% ^>^> "Z:\new_files.log" >> "C:\scripts\comskip\trans.bat"
ECHO TASKKILL /IM vlc.exe >> "C:\scripts\comskip\trans.bat"
ECHO Rem Batch commands for %1 FINISH >> "C:\scripts\comskip\trans.bat"

Thus this script creates the trans.bat file, which is then run via a scheduled task. All the stuff starting with REM are lines that the script no longer uses, but that contain interesting code that might be useful in the future. This includes using a program called mpeg2repair, that was meant to clean the recording in-case there were any errors during transmission, but the program error-ed out too many times. Another line was also meant to use Mencoder to cut the adverts out directly using an EDL file created by comskip, but this resulted too often in audio/video sync problems, I think as a result of the original .ts file having some errors in it, so I REMed out that line as well.

The second script is my Ubuntu bash script, which is run every morning, after the trans.bat script (created by the script above) is run.

#!/bin/bash
# Written by FakeOutdoorsman and updated by mysoogal further updated by prupert
# Attribution-Noncommercial 3.0 Unported
# http://creativecommons.org/licenses/by-nc/3.0/deed.en
# trackback http://ubuntuforums.org/showthread.php?t=960627
echo trans.sh started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.ts" >> /mnt/media/documents/ruperts/TV/trans.log
chmod -R 777 /mnt/media/documents/ruperts/TV
find /mnt/media/documents/ruperts/TV -name "*.ts" -exec /home/server/Desktop/handbrake/HandBrakeCLI -i {} -t 1 -c 1 -o {}.mkv -f mkv -e xvid -b 1200 -2  -a 1 -E faac -B 160 -R 0 -6 dpl2 -D 1 -C 2  -v '{}' \; >> /mnt/media/documents/ruperts/TV/trans.log
echo trans.sh finished on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
echo attempting to add advert chapter markers on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec /home/server/scripts/mkvmerge.sh "{}" \; >> /mnt/media/documents/ruperts/TV/trans.log
echo advert chapter markers added on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
echo starting clear up on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
cd "/mnt/media/documents/ruperts/TV/" || { echo >&2 "Source folder not found"; exit 1; } >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts" -exec sh -c '[ -f "${1%.ts}.ts.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts" -exec sh -c '[ -f "${1%.ts}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts.mkv" -exec sh -c '[ -f "${1%.ts.mkv}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts.chap" -exec sh -c '[ -f "${1%.ts.chap}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec mv {} /media/video/Video \; >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.edl" -exec rm {} \; >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.txt" -exec rm {} \; >> /mnt/media/documents/ruperts/TV/trans.log
echo ts files deleted and moved script stopped on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log

This is run by a cron job every morning. It uses Handbrake to transcode the .ts file to a .mkv file. It then runs a third script (below called mkvmerge.sh) to use mkvmerge to add chapter markers to the mkv file that Handbrake created. Finally, it moves the mkv file to where all my Videos are stored. At the very end, there are some commands to delete the .ts files and the intermediate .mkv files ONLY if the mkv file has been correctly created.

Here is the mkvmerge.sh script:

#!/bin/bash
echo mkvmerge.sh started on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
VIDEO=$1
ENC=${VIDEO##*/}
mkvmerge --chapters "${VIDEO%/*}/${ENC%%.*}.ts.chap" "$VIDEO" -o "/mnt/media/documents/ruperts/TV/${ENC%%.*}.mkv"

This script is called by the main Ubuntu trans.sh script above.

I also have a separate script that I use to transcode the .m2ts files produced by the Sony PS3′s PlayTV program:

#!/bin/bash
echo transm2ts.sh started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/media/documents/ruperts/TV/transm2ts.log
find /mnt/media/documents/ruperts/TV -name "*.m2ts" >> /mnt/media/documents/ruperts/TV/transm2ts.log
chmod -R 777 /mnt/media/documents/ruperts/TV
find /mnt/media/documents/ruperts/TV -name "*.m2ts" -exec /home/server/Desktop/handbrake/HandBrakeCLI -i {} -t 1 -c 1 -o {}.mkv -f mkv -w 720 -e xvid -b 1200 -2 -a 1 -E faac -B 160 -R 0 -6 dpl2 -D 1 -C 2 -v '{}' \; >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo transm2ts.sh finished on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo starting clear up on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
cd "/mnt/media/documents/ruperts/TV/" || { echo >&2 "Source folder not found"; exit 1; } >> /mnt/media/documents/ruperts/TV/transm2ts.log
find . -type f -name "*.m2ts" -exec sh -c '[ -f "${1%.m2ts}.m2ts.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/transm2ts.log
find . -type f -name "*.m2ts" -exec sh -c '[ -f "${1%.m2ts}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/transm2ts.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec mv {} /media/video/Video/TV \; >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo m2ts files deleted and moved script stopped on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
exit

Because Comskip can’t be run on .m2ts files, I don’t call the mkvmerge.sh script, since there is no .chap file to use.

I would freely admit that these scripts aren’t very user friendly, since they are customised just for my setup. However, if readers would prefer a useful version that was more user-friendly, I could create that. It is possible to convert the Ubuntu Bash script into a Bat script for Windows. However, I don’t think there is any alternative to the mkvmerge file for Windows that the Ubuntu script uses.

Also, sadly, the devs of Handbrake are a little blinkered. They created the CLI interface so you can run Handbrake via the commandline, which is great. The CLI also creates Exit codes, which is very useful for scripting. However, the Exit codes only tell you if the CLI worked correclty, not if Handbrake worked correctly. So if the transcode failes, but the CLI worked corretly, it will exit as if nothing went wrong. Sadly the Handbrake devs don’t see this as a problem. Kind stupid imho, but hey, it’s their loss. Thus, there is no way of telling if the transcode worked correctly or not, the only way to check is to see if the file you want to be created is there…doh!

Turn programs on and off with Bluetooth in Ubuntu

Having recently been messing around with ZoneMinder for home security, I wanted to turn on and off the program using Bluetooth. Basically, the idea being that when I am around, the server will sense my presence (no Darth Vadar jokes please ) via the Bluetooth on my phone and turn off ZoneMinder. Then, if I leave the location of the server, it will no longer be able to sense Bluetooth on my phone and thus will turn on ZoneMinder.

So, I hunted around for such a program, I came across bluemon and BlueProximity, but they both only worked on proximity, i.e. they relied on you being near by, as far as I could tell, there was no option of running an action when the phone was completely unavailable.

So, I decided to write my own script to do what I wanted. Now, bear in mind I have very little skill when it comes to Bash and scripting and what I wrote was a result of trial and error. It all revolves around a tool called hcitool that can be used in Linux to control Bluetooth connections.

The first attempt at my script can be found here and with another link below.

It is only designed to work with ZoneMinder and there isn’t much advice within the code, but I shall update it as time goes by to make it more useful

Here is the full link to the script:

bluetoothscan

Tags: bluetooth, Code, Scripts, ubuntu

How to manage your time with Windows Mobile

If you are like me and are a Windows Mobile user, you might feel a little left in the shadows by all the iPhone love out there. I use the HTC Vox (actually Orange’s SPV E650) and find it a great little app. It’s got a slide out keyboard and is nice and small.

SPV E650

I am always trying out new ways to best use this device, so I thought I would share what I currently have settled on. First off, a little background about what I want to achieve from this device. I, like most normal users, am stuck behind a desk most of the time, I am not a road warrior, so I only need my device to help me out, not run my life. I mainly use it to  manage my time and make notes, not as my only work tool. Thus, I have concentrated on time management tools.

My main task management tool is WebIS’ PocketInformant, a replacement to the default Pocket Outlook installed on my phone. PocketInformant has a huge amount of options and allows you to define different calender views, add reminders about meetings, tasks and notes, as well as manage your contacts. My main use is to simply add any reminders that I need about project and task due dates, and also write notes about projects I am working on, alongside reminders for these notes.

PocketInformant

My next tool is SBSH’s Facade. This is a home screen plugin that alongside adding shortcuts to as many apps as you want, meters for various phone stats, it also adds a handy calender preview right on the home screen. You can use it to instantly check up on task that day, simply by unlocking the phone to operate the screen. I find it incredibly handy.

Facade

Finally, as a bonus, if you also have SBSH’s Pocket Weather, you get a weather forecast in your home screen to

Pocket Weather

Finally, to help me manage all my thoughts and ideas I have when I am on the move, I simply whack them down into the phone’s built in notes application – or Pocket Word depending on how big my blabbing will be. Then, I transfer over the file and copy and paste into a tiddlywiki file. Not the most elegant of solutions and I could edit the tiddlywiki file on the phone, but it seems like PIE isn’t designed for such a java heavy page.

So, this is how I work and it works great for me.

Tags: How To, windows mobile

Filter posts from RSS feeds

Ever had any posts from a feed you wish you didn’t have to read through (I know, the pain the heart ache, you have to actually scroll through a post you don’t like, man, we are so lazy us modern humans)? For example all the incessant drivelling about iPhone apps, or annoying adds in feeds, such as the ones that lifehacker have started to add, like the one below:

lifehacker feed add

Now, if you are like me and find these adds annoying, you can get rid of them. There are a number of possibilities, but so far only one works at the moment, the rest might take time to work.

If you use Google’s Feed Reader and Firefox, you can try the Google Reader Filter for the Greasemonkey extension (and as mentioned by lifehacker themselves). This allows you to add filter words so that posts containing them are greyed out, the advantage being you still have the post to read if you want. However, I couldn’t get it to work and it only works in the list view. In fact, some comments on the scripts page indicate that it doesn’t work in Firefox 3 and that the script author is no longer maintaining it, so this might be a no-starter.

Next, you can try to write a Yahoo Pipes filter, such as this one here that I created. However, it doesn’t seem to work for me . This is more than likely due to my naff programming skills, you might have better luck.

Finally, there are two web services you can use, FilterMyRSS and FeedRinse. I haven’t had any results from FilterMyRSS yet, but so far FeedRinse has worked wonders. I simply added the full lifehacker RSS feed, then added a filter to block posts that contain [Sponsored] in the title. Then, subscribe to the feed in your favourite reader, and tada, gone are those annoying adds. Of course, you might have to update the filter based on how lifehacker changes how they present these adds Of course, you can add any other term to filter out, like “iPhone” to remove all the iPhone drivel that fills lifehacker’s pages these days. The disadvantage with this solution is, you can’t see what feeds have been filtered out, so you don’t know if you are missing loads of posts that you don’t actually want to be filtered out. But hey, at least the adds are gone

Tags: How To, lifehacker, RSS

More backup goodness with rsync

So, a while ago I posted about how to use rsync and rsback to backup important files and folders. Wanting to be extra safe, I decided to create another backup on a separate disk stored somewhere else at home. This will act as a form of redundant backup. It wont backup everything that often, but I will always have something to fall back on in case both my main drive and backup drive fail, which is quite unlikely.

NSLU2

This solution uses the rather fabulous little NSLU2, a great little NAS device made by Linksys, I will post more about this little tool later.

Once again I decided to make use of rsync, since it seems like the best option and I was now more comfortable with it. I wanted my NSLU2 to pull data from my Ubuntu server onto a USB disk attached to the NSLU2. The best way to do this, is run an rsync daemon on the server and run rsync on the NSLU2 scheduled via cron.

First, you need to install rsync on the server, which I had already done for my previous backup solution, use synaptics and search for rsync if you haven’t got it installed already. Next, to get rsync to run in daemon mode automatically, you need to edit the /etc/default/rsync file and ensure that

RSYNC_ENABLE=true

Next, you need to create an rsyncd.conf file, located at /etc/, that controls how the daemon will operate. Now, you have to be carefull here with permissions, I had loads of problems with permissions before I figured this one out. You can set a number of options in the rsyncd.conf file to do with security and what folders can be accessed. Here is mine:

uid = root
gid = root
motd file = /etc/rsyncd.motd
log file = /etc/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock

[backup]
path = /mnt/media/backupsfolder
comment = Media Disk Backup
uid = nobody
gid = nobody
read only = yes
list = yes
hosts allow = 192.168.1.142

The most important part is uid and gid, which determines what user the daemon runs at, whichever you choose, that user MUST have the correct permission to write to the directory that the log and pid files are stored. If you don’t choose a user with the correct permission, you will find that the daemon will run, but wont be able to transfer anything. The motd location and file is for the Message Of The Day, the file can contain whatever little message you want the rsync daemon to show when the rsync client connects. [backup] and what follows lists the folders that you want to make available for copying. You can have as many different folders as you want, each with their own [name] and file location. You can also change which rsync clients are allowed to read which file lists, using the hosts allow field. There are other options you can add here, to do with users and passwords, but I haven’t bothered since no one else on my network is using rsync.

Now that that is set up, you then need to set up the actual rsync client that does the pulling of the files on the NSLU2. I have decided to do this by a script that is run via cron. So, go create a script somewhere, I have stored mine at /etc/rsync/backuprsync.sh. When you create the file, make sure you make it executable, I chose chmod 744 /etc/rsync/backuprsync.sh. I wanted to have some details about the rsync process logged to a file, that was emailed to me, so, I added some extra code to the script. Here it is:


#!/bin/sh
echo Subject: Backup Backup Started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/disk/backuprsync.log
rsync -av --stats rsync://192.168.1.111/backup/ /mnt/disk/backup/ >> /mnt/disk/backuprsync.log
echo END: Backup Backup Complete on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/disk/backuprsync.log
cat /mnt/disk/backuprsync.log | ssmtp someaddress@gmail.com

What this does is first removes rm the old logfile (so make sure when you first run this, there is a log file there, otherwise the script will fail It then prints the time the backup started echo Subject:... to some location I have chosen /mnt/disk/backuprsync.log. Next is the actual rsync command. The -av --stats options creates an archive file (a), creates detailed output (v) and creates some stats at the end (stats). rsync://192.168.1.111/backup/ is the address for the Ubuntu server based on its IP address, whilst backup is the name of the file location set up in the rsyncd.conf file. If you want to change the backup job to another file location, then just change backup to whatever you called the other job. /mnt/disk/backup/ is the location on the NSLU2 where the backup is stored. Finally, >> /mnt/disk/backuprsync.log will output the results of -v and –stats to the log file (by the way using >> instead of > means it appends the output, instead of overwrites it).

The next echo command prints when the job has ended to the end of the log file, as before (note the use of >> to append, rather than overwrite). Finally, cat /mnt/disk/backuprsync.log | ssmtp someaddress@gmail.com takes the log file and emails it to me using ssmtp. I had to use this as the OS I am using on the NSLU2 doesn’t have its own mail command. There will be more about setting up ssmtp in another post.

Finally, you’ll want to run this job via cron, so, add

01 1 3 * * root /etc/rsync/backuprsync.sh

To your /etc/crontab file. Ensure that your crontab file knows to use the sh shell, so it should have SHELL=/bin/sh at the beggining of your crontab file. This crontab entry makes cron run the script at 01.13am on the 3rd day of every month.

If all goes to plan, rsync will connect to your server’s rsync daemon, suck all the changed files from last time to the NSLU2 and then email you the results. Sweet.

Tags: backup, How To, NSLU2

Setting up virtual hosts with Apache

Thought I should post about setting up virtual hosts in Apache, since this was one of the first things I had to do. I got all this info from this site, which was incredibly useful. For all the following instructions, when you see these instructions:

enter some/text/here

You need to enter this text exactly as you see it in your terminal, which can be found at Applications>Accessories>Terminal

First, you need to set up your folders where the sites are gonna be stored:

mkdir /home/www/www.example.com
mkdir /home/www/www.example.net

Then you need to tell Apache that you are gonna use virtual hosts. Create a file using:

sudo touch /etc/apache2/conf.d/virtual.conf

You need to add the following content, use:

sudo gedit /etc/apache2/conf.d/virtual.conf

and enter the following:

#
#  We're running multiple virtual hosts.
#

NameVirtualHost *

Now, you have to create an entry in both the sites-enabled and sites-available directories in /etc/apache2. So, first create the following file:

sudo touch /etc/apache2/sites-available/www.example.com

and enter the following by using:

sudo gedit /etc/apache2/sites-available/www.example.com

and enter:

#
#  Example.com (/etc/apache2/sites-available/www.example.com)
#
<VirtualHost *>
        ServerAdmin webmaster@example.com
        ServerName  www.example.com
        ServerAlias example.com

        # Indexes + Directory Root.
        DirectoryIndex index.html
        DocumentRoot /home/www/www.example.com/htdocs/

        # CGI Directory
        ScriptAlias /cgi-bin/ /home/www/www.example.com/cgi-bin/
        <Location /cgi-bin>
                Options +ExecCGI
        </Location>

        # Logfiles
        ErrorLog  /home/www/www.example.com/logs/error.log
        CustomLog /home/www/www.example.com/logs/access.log combined
</VirtualHost>

Next, do the same for your second site, so create a file using:

sudo touch /etc/apache2/sites-available/www.example.net

and enter the following by using:

sudo gedit /etc/apache2/sites-available/www.example.net

and enter:

#
#  Example.net (/etc/apache2/sites-available/www.example.net)
#
<VirtualHost *>
        ServerAdmin webmaster@example.net
        ServerName  www.example.net
        ServerAlias example.net

        # Indexes + Directory Root.
        DirectoryIndex index.html
        DocumentRoot /home/www/www.example.net/htdocs/

        # CGI Directory
        ScriptAlias /cgi-bin/ /home/www/www.example.net/cgi-bin/
        <Location /cgi-bin>
                Options +ExecCGI
        </Location>

        # Logfiles
        ErrorLog  /home/www/www.example.net/logs/error.log
        CustomLog /home/www/www.example.net/logs/access.log combined
</VirtualHost>

Finally, you need to create a copy of these files in the sites-enabled directory, using what is called a symbolic link. You can use a funky little program called a2ensite to do this. So, enter the following:

sudo a2ensite www.example.com
sudo a2ensite www.example.net

And finally (I was lying last time), you need to restart Apache for this to all take effect:

sudo /etc/init.d/apache2 reload

And tada, it should all be working, yay!

Tags: apache, How To

Setting up WordPress

Formatting and Structure

One interesting thing I just noticed. When you write a page, and presumably a post, and WordPress creates a permalink for you, make sure you know what you want to call it, and don’t change it. It seems that wordpress uses the first title you choose as the permalink title. Even if you change the title later, it uses the original title as the permalink. Something to bear in mind when you get home late, drunk, and decide to write a rant about curbs and giving it a less than gregarious title…Ooh, I’ve just noticed you can edit it, in the permalink option beneath your page title..very cool.

Tags: blog, How To, wordpress