pruperting

MythTV Recordings: Rename, Move and Delete From The Database

So, I have been using MythTV for a while now and it is a.w.e.s.o.m.e.. However, I had one problem, since I transcode and store all my videos on my Ubuntu server and not my MythTV box (which is off most of the time via mythwelcome) I had to manually transfer recorded files to my Ubuntu server via mythweb.

I thought this was a bit of a hokey solution, so I asked around if there was a script that allowed you to rename MythTV recordings (to a more human readable version), move them to another location and remove them from the MythTV database. There were some solutions out there, but they all involved the recorded files being kept in the MythTV database.

So, I wrote my own script. This script renames the MythTV files to a nice human understandable version using mythrename.pl, it then moves the recordings to a folder of your choice (in my case a mounted samba share on my server), it then uses myth.find_orphans.pl to delete the moved recordings from the mythTV database (I don’t know if it will still remember that these files were previously recorded so it doesn’t rerecord them or not, I guess time will tell).

So, without further ado, here is the script, in lovely Bash (the script is called mythtvremodel (mythtv REname MOve DELete, geddit?):


#! /bin/sh
### BEGIN INIT INFO
# Provides: mythtvremodel
# Required-Start:    $local_fs $syslog $remote_fs dbus
# Required-Stop:     $local_fs $syslog $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: rename, move delete recordings
### END INIT INFO
# script to rename files recorded from mythtv and then to move them and delete them
#made by prupert.co.uk
#creative commons licence, whichever is the most FREE (as in beer)

#lets put in some variables so others can use this
#location of log file, you can choose anything and any name
log=/path/to/mythtvremodel.log
#this is the full path to mythrename.pl
mrename=/location/to/mythrename.pl
#this is the full path to your recordings folder
record=/var/www/mythweb/data/recordings/
#this is the extension your recordings are stored as, usually mpg or nuv
ext=mpg
#this is the full path to where you want your files moved to
dest=/path/to/folder/to/move/recordings/to/
#this is the full path to myth.find_orphans.pl
morphan=/path/to/myth.find_orphans.pl
#mythtv database password for myth.find_orphans.pl
pass=yourmythtvdatabasepassword

# first, run mythtvrename to get nice names for the recordings
echo starting mythtvremodel on `date "+%m/%d/%y %l:%M:%S %p"` > $log
perl $mrename --underscores --format %T-%S-%d%m%y >> $log
# use find to search for all recorded files in the recording folder and move them to the TV folder
find "$record" -name "*.$ext" >> $log
find "$record" -name "*.$ext" -exec mv {} "$dest" \; >> $log
# use mythorphan to remove the now missing recorded files from the database
perl $morphan --pass=$pass --dodbdelete >> $log
#that should be it
echo stopping mythtvremodel on `date "+%m/%d/%y %l:%M:%S %p"` >> $log
exit 0

And there you have it. Now, since I run mythbuntu, I have put this script in /etc/rc0.d (thats rc zero . d) and have named it K03mythtvremodel, thus it is run when my system shuts down, but before all the samba shares and drives are unmounted. As I said, since I use mythwelcome to shutdown my machine when idle, this seemed like the best solution.

Hope it helps someone!

MediaCenter: PS3 No, XBMC YES!

So, for many months I have been slaving away with my PS3 and server, getting Mediatomb to work well and finding a setting for ffmpeg that my PS3 will like. What a waste of time!!

The fact that the PS3 only supports UPnP or locally stored videos is a joke, I mean, come on Sony, Windows Shares are where it is at. Mediatomb is great at what it does, but the interface of any UPnP device is just SO horrible.

However, the interface on XBMC is just liquid gold, that can be further customised via skins like Aeon or Mediastream. Both look truly beautiful and put the PS3 to shame. I would dearly love to run XBMC on my old Xbox, but sadly it just hasn’t got the processing power to play my .x264 encoded mp4 videos, so I will use it just as my Music player (still using XBMC on it though). Thus my HTPC, which primarily runs MythTV will now also be used to view my videos via XBMC. It seems rediculous to me that a machine as powerful as the PS3 comes with such an ugly and limited Media Player interface as the UPnP protocol. I guess because Sony are shortly to release Video Store to stream and download Videos, they want that to be the main way PS3 users access videos, and not ones they already own. At the end of the day, it is all about new revenue streams

Still, all I can say is that XBMC rules, and I wish I had an Acer Revo with the new Nvidea Ion platform to truly appreciate the full power of XBMC without worrying about messing up my MythTV set-up

How To Use FFMPEG To Transcode Video For The PS3

So, it seems my last post actually wasn’t the best solution. Although using Mediatomb and VLC worked some of the time, often I found that the audio and video was out of sync, making it impossible to watch the movie.

So, I looked for another alternative. I already use HandBrakeCLI to transcode my videos, but that is essentially a front end to ffmpeg and x264. They are moving away from providing decent support for transcoding videos on the Playstation 3, so I decided to use ffmpeg directly. I used this guide to install the latest svn versions of ffmpeg and x264 on my Ubuntu 8.04 server. I then spent about two weeks trying to find an up-to-date command to actually use ffmpeg to create a PS3 compatible file. It seems most of the instructions on the web are way out of date and use commands that ffmpeg no longer accepts. Finally, after lots of trying I finally asked on the IRC channel #ffmpeg. This very nice dude called relaxed suggested the following:

ffmpeg -t 120 -i $input -vcodec libx264 -level 41 -vpre hq -crf 24 -threads 0 -acodec libfaac -ab 128k -ac 2 output.mp4

Sadly I had problems with this code, since the .mpg files I was using (I am now using MythTV instead of MediaPortal, but that’s another post) were not accepted by the libfaac codec.

However, with a little modification, I got the perfect command. So, without further ado, the best command I have found to convert ANY video accepted by ffmpeg into a PS3 compatible file using x264 that is good quality and low file size is:

ffmpeg -y -i "$1" -vcodec libx264 -level 41 -vpre normal -crf 24 -threads 0 -acodec libfaac -ab 128k -ac 2 -ar 48000 "$1".mp4

Obviously change $1 into your input file and “$1″.mp4 into your output file. If you want to send the command line output to a log file so you can see what is going in, use:

ffmpeg -y -i "$1" -vcodec libx264 -level 41 -vpre normal -crf 24 -threads 0 -acodec libfaac -ab 128k -ac 2 -ar 48000 "$1".mp4 > /where/you/store/your/logs/ffmpegprog.log 2>&1

Now, when I first saw this I thought, hey, its got hardly any switches (all the -XYZ stuff), therefore it must be cack.  However, the key is the “-vpre normal” switch.  This tells ffmpeg to use some of its built-in presets, or in this case built-in x264 presets that are created when you build x264 from SVN. I was initially told, as you can see from above, to use the “hq” preset, but I found this resulted in files that were quite big. Using the “normal” preset seemed to produce files that were of great quality compared to my original source (SD DVB-T transport streams in MPG format) but much smaller files size, for example, a .mpg transport stream movie that is originally 2.5GBs in size gets reduced to around 650 MBs, awesome.

So, this is the command I now use to convert all my videos. I also use comskip run via Wine to check for commercials and mp4box to mux in the chapter markers from comskip. Weirdly, I do find that when playing these resultant files via VLC they appear to have their audio and video out of sync, but on the PS3 they are fine. Whether this is due to me using an old VLC or my PC being too slow to play em, I don’t know, but hey, if it works.

So, if you are looking for the best ffmpeg command to transcode video files for the PS3 using ffmpeg and x264, I am pretty sure this is it.

As an aside, I also built HandBrakeCLI from SVN and it now also has a basic preset, called Normal, that also produces PS3 compatible files. I will play with them to see how they do, though I find that occasionally HandBrake can crash, and get stuck, so I try to avoid it. Sadly though, the HandBrake dev’s wont help you with this, since you are using the HandBrakeCLI and it is not the CLI that is crashing so they wont look into it for you. In fact (<rant>) I am really shocked at the rudeness and arrogance that the HandBrake devs demonstrate on their forum. I have read so many well worded and polite posts from confused people on their forums, where the devs (mainly a guy called jbrake) just answers back incredibly sarcastically and very rudely, providing little to no help. Maybe they get loads of annoying whiny posts from people who haven’t RTFM, but still, there is no need to be rude about it. In fact, that is the main reason I don’t want to use HandBrake, since I don’t like the attitude those guys have (</rant>).

Anyway, hope this awesome command for a recent SVN ffmpeg and x264 version to convert video files for PS3 works for you!

UPnP, PS3, Transcoding and MediaTomb

For quite a while I had been using PS3MediaServer to provide access to my Video files from my PS3. The problem was, I was getting huge problems getting it to support anything other than the files that the PS3 natively supports. I had previously messed around with MediaTomb but I couldn’t get it’s transcoding features to work. But then I came across this post and it sorted all my problems out. I simply followed the tutorial there and made some minor adjustments, which I will describe below.

MediaTomb

The quickest way to follow the tutorial is to cheat and just copy and paste the config.xml that is kindly provided. However, I have found that copy and pasting the whole config.xml doesn’t work. I am pretty sure it is to do with the <udn>uuid= part of the config, since I guess this aligns and individual config to an individual MediaTomb installation. Thus, simply copy and paste all the stuff after and including this line:

“<protocolInfo extend=”yes”/><!– For PS3 support change to “yes” –>

and you’ll be good to go. Next, since most of my files are in the MKV container, I needed to add some extra stuff to support MKV files and the transcoding of them. So, I added the following two lines to their respective places (you can tell where they should go by looking at their format and looking for other lines that look similar). I added these two lines:

<map from=”mkv” to=”video/x-matroska”/>

<transcode mimetype=”video/x-matroska” using=”vlcmpeg”/>

And awesomely, it worked straight out of the box. Nice one Mr Pyke! There are a few more things to add, like adding thumbnail support, which I understand you can do by using a script and app called thumbnailermpeg, but that’s about it. I have found that using MediaTomb and VLC to transcode files puts much less strain on my server than using PS3MediaServer ever did. HTOP shows MediaTomb and VLC when streaming an MKV file pushing the server to around 15% CPU usage on my dual core Pentium4. Doing the same thing for one of the few supported movies by PS3MediaServer shows CPU usage maxed out at 100% on both cores full time. Ouchy! So I have saved some energy and prolonged the lifespan of my server by doing this too. I know that at the moment MediaTomb doesn’t do all the fancy things that PS3MediaServer does (like AC3 pass-through and stuff) but I can live without that. I am sure there was a lot I was missing in PS3MediaServer that it can do and it is updated much more frequently than MediaTomb, but I just couldn’t agree with the huge resource usage it entails.

I now have full streaming of all my recorded-off-the-tele MKV movies and I am a happy bunny, so thanks RobertPyke and thanks MediaTomb!

MediaPortal Comskip and Transcode Bat and Bash Scripts

As I mentioned earlier, I use MediaPortal to record DVB-T broadcasts. As these end up quite big, I then use two scripts to check the recordings for adverts and to transcode the recordings to a mkv file (smaller and better supported).

This solution isn’t very elegant, since it is for my personal setup which is a little unusual, in that the recordings are done via a Windows XP MediaCentre PC, they are then analysed for adverts using Comskip and are then transferred to my Ubuntu-based server, which then transcodes the files to an MKV file and the adds Chapter markers to those MKV files using the file from Comskip. Thus there script is in two parts, one a Windows Bat script and the other is a Ubuntu Bash script.

The scripts are below. First, the Windows Bat script that runs the recorded file through Comskip and then copies it over to the server. Since I wanted to only run one script for all the recordings made in an evening, the script takes the file name from MediaPortal, which passes it to the script after the file is finished recording, using the Comskip plugin. This explains all the weird “Echo” in front of all the terms, since basically it is putting all the commands into another script, which is then run early each morning.

BASH

@ECHO OFF
ECHO ECHO.
ECHO Rem Batch commands for %1 START >> "C:\scripts\comskip\trans.bat"
ECHO CD "C:\scripts\comskip" >> "C:\scripts\comskip\trans.bat"
ECHO START "" "C:\Program Files\VideoLAN\VLC\vlc.exe" >> "C:\scripts\comskip\trans.bat"
ECHO %%SystemRoot%%\explorer.exe "Z:\" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO ************************** ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Post Processing started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO Starting mpeg2repair %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO mpeg2repairHelper.exe %1 %1-fixed.ts >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO mpeg2repair finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Comskip started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO comskip %1 >>"%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Comskip finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO %%date%% %%time%% ^>^> "Z:\new_files.log" >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO ComClean started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO mencoder "%~dpn1.ts" -edl "%~dpn1.edl" -alang eng,nar -noskip -vf harddup -oac copy -ovc copy -of mpeg -o "%~dpn1_clean.ts" >> "C:\scripts\comskip\trans.bat"
REM ECHO ECHO ComClean finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO New file %1 being moved to Server ^>^> "Z:\new_files.log" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Tidying started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO %%SystemRoot%%\explorer.exe "Z:\" >> "C:\scripts\comskip\trans.bat"
ECHO CD "%~dp1" >> "C:\scripts\comskip\trans.bat"
REM ECHO ren *.edl *.ts.edl >> "C:\scripts\comskip\trans.bat"
REM ECHO ren *.txt *.ts.txt >> "C:\scripts\comskip\trans.bat"
ECHO ren *.chap *.ts.chap >> "C:\scripts\comskip\trans.bat"
ECHO ECHO Moving Clean TS to Server directory on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
REM ECHO IF EXIST "%~dpn1_clean.ts" del %1 >> "C:\scripts\comskip\trans.bat"
ECHO xcopy %1 Z:\ >> "C:\scripts\comskip\trans.bat"
ECHO IF EXIST "Z:\%~n1.ts" del %1 >> "C:\scripts\comskip\trans.bat"
REM ECHO xcopy "%~dpn1.edl" Z:\ >> "C:\scripts\comskip\trans.bat"
ECHO del "%~dpn1.txt" >> "C:\scripts\comskip\trans.bat"
ECHO xcopy "%~dpn1.ts.chap" Z:\ >> "C:\scripts\comskip\trans.bat"
ECHO ECHO TS and others moved to Server directory on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:\scripts\comskip\trans.bat"
ECHO ECHO %~n1.ts moved to Server on %%date%% %%time%% ^>^> "Z:\new_files.log" >> "C:\scripts\comskip\trans.bat"
ECHO TASKKILL /IM vlc.exe >> "C:\scripts\comskip\trans.bat"
ECHO Rem Batch commands for %1 FINISH >> "C:\scripts\comskip\trans.bat"

Thus this script creates the trans.bat file, which is then run via a scheduled task. All the stuff starting with REM are lines that the script no longer uses, but that contain interesting code that might be useful in the future. This includes using a program called mpeg2repair, that was meant to clean the recording in-case there were any errors during transmission, but the program error-ed out too many times. Another line was also meant to use Mencoder to cut the adverts out directly using an EDL file created by comskip, but this resulted too often in audio/video sync problems, I think as a result of the original .ts file having some errors in it, so I REMed out that line as well.

The second script is my Ubuntu bash script, which is run every morning, after the trans.bat script (created by the script above) is run.

#!/bin/bash
# Written by FakeOutdoorsman and updated by mysoogal further updated by prupert
# Attribution-Noncommercial 3.0 Unported
# http://creativecommons.org/licenses/by-nc/3.0/deed.en
# trackback http://ubuntuforums.org/showthread.php?t=960627
echo trans.sh started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.ts" >> /mnt/media/documents/ruperts/TV/trans.log
chmod -R 777 /mnt/media/documents/ruperts/TV
find /mnt/media/documents/ruperts/TV -name "*.ts" -exec /home/server/Desktop/handbrake/HandBrakeCLI -i {} -t 1 -c 1 -o {}.mkv -f mkv -e xvid -b 1200 -2  -a 1 -E faac -B 160 -R 0 -6 dpl2 -D 1 -C 2  -v '{}' \; >> /mnt/media/documents/ruperts/TV/trans.log
echo trans.sh finished on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
echo attempting to add advert chapter markers on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec /home/server/scripts/mkvmerge.sh "{}" \; >> /mnt/media/documents/ruperts/TV/trans.log
echo advert chapter markers added on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
echo starting clear up on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
cd "/mnt/media/documents/ruperts/TV/" || { echo >&2 "Source folder not found"; exit 1; } >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts" -exec sh -c '[ -f "${1%.ts}.ts.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts" -exec sh -c '[ -f "${1%.ts}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts.mkv" -exec sh -c '[ -f "${1%.ts.mkv}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts.chap" -exec sh -c '[ -f "${1%.ts.chap}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec mv {} /media/video/Video \; >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.edl" -exec rm {} \; >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.txt" -exec rm {} \; >> /mnt/media/documents/ruperts/TV/trans.log
echo ts files deleted and moved script stopped on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log

This is run by a cron job every morning. It uses Handbrake to transcode the .ts file to a .mkv file. It then runs a third script (below called mkvmerge.sh) to use mkvmerge to add chapter markers to the mkv file that Handbrake created. Finally, it moves the mkv file to where all my Videos are stored. At the very end, there are some commands to delete the .ts files and the intermediate .mkv files ONLY if the mkv file has been correctly created.

Here is the mkvmerge.sh script:

#!/bin/bash
echo mkvmerge.sh started on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
VIDEO=$1
ENC=${VIDEO##*/}
mkvmerge --chapters "${VIDEO%/*}/${ENC%%.*}.ts.chap" "$VIDEO" -o "/mnt/media/documents/ruperts/TV/${ENC%%.*}.mkv"

This script is called by the main Ubuntu trans.sh script above.

I also have a separate script that I use to transcode the .m2ts files produced by the Sony PS3’s PlayTV program:

#!/bin/bash
echo transm2ts.sh started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/media/documents/ruperts/TV/transm2ts.log
find /mnt/media/documents/ruperts/TV -name "*.m2ts" >> /mnt/media/documents/ruperts/TV/transm2ts.log
chmod -R 777 /mnt/media/documents/ruperts/TV
find /mnt/media/documents/ruperts/TV -name "*.m2ts" -exec /home/server/Desktop/handbrake/HandBrakeCLI -i {} -t 1 -c 1 -o {}.mkv -f mkv -w 720 -e xvid -b 1200 -2 -a 1 -E faac -B 160 -R 0 -6 dpl2 -D 1 -C 2 -v '{}' \; >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo transm2ts.sh finished on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo starting clear up on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
cd "/mnt/media/documents/ruperts/TV/" || { echo >&2 "Source folder not found"; exit 1; } >> /mnt/media/documents/ruperts/TV/transm2ts.log
find . -type f -name "*.m2ts" -exec sh -c '[ -f "${1%.m2ts}.m2ts.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/transm2ts.log
find . -type f -name "*.m2ts" -exec sh -c '[ -f "${1%.m2ts}.mkv" ]' _ {} \; -delete >> /mnt/media/documents/ruperts/TV/transm2ts.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec mv {} /media/video/Video/TV \; >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo m2ts files deleted and moved script stopped on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
exit

Because Comskip can’t be run on .m2ts files, I don’t call the mkvmerge.sh script, since there is no .chap file to use.

I would freely admit that these scripts aren’t very user friendly, since they are customised just for my setup. However, if readers would prefer a useful version that was more user-friendly, I could create that. It is possible to convert the Ubuntu Bash script into a Bat script for Windows. However, I don’t think there is any alternative to the mkvmerge file for Windows that the Ubuntu script uses.

Also, sadly, the devs of Handbrake are a little blinkered. They created the CLI interface so you can run Handbrake via the commandline, which is great. The CLI also creates Exit codes, which is very useful for scripting. However, the Exit codes only tell you if the CLI worked correclty, not if Handbrake worked correctly. So if the transcode failes, but the CLI worked corretly, it will exit as if nothing went wrong. Sadly the Handbrake devs don’t see this as a problem. Kind stupid imho, but hey, it’s their loss. Thus, there is no way of telling if the transcode worked correctly or not, the only way to check is to see if the file you want to be created is there…doh!