TV show commercial cutting – Part 2

I’ve had a few people ask about my Applescript which I use to post-process TV show recordings created by my Elgato EyeTV HD encoder, so here’s some more details.

First, I have this basic configuration:

  1. Comcast digital signal fed into a Comcast RG110 digital set top box
  2. Video/Audio component output from the Comcast set top box are fed into an Elgato EyeTV HD encoder
  3. EyeTV HD encoder is connected to my Apple iMac desktop computer via a USB 2.0 cable
  4. The EyeTV3 software on the iMac controlling the recording of scheduled TV shows
  5. IR Changer (Blaster) connected to the Comcast set top box allowing the EyeTV HD encoder to change the TV stations appropriately

I also have installed the freeware tool called ETVComskip which is used to identify commercial segments in a recorded MPEG file and saves them to a text file (with an .edl extension). This is the primary tool that I use for commercial detection, so I’ve spent a lot of time tuning the settings for this application to get good commercial detection (which I’ll describe in a future posting).

Luckily, the EyeTV3 software is able to run two specific Applescripts when a recording has just started (RecordingStarted.scpt) and when it has finished (RecordingDone.scpt). When you install the ETVComskip application it will automatically modify both of these scripts to support the commercial detection process.

Now, the ETVComskip tool is basically a Python script that uses a program called Comskip to detect the commercials in an MPEG video file. The Comskip app is actually a Microsoft Windows application, so the Python script is setup to use Wine (WIN32 emulator) to run the Comskip windows executable on your Mac. You don’t need to worry about all of this, as the makers of ETVComskip have taken care of everything. The only thing to note, is that you will see the X-Windows App icon appear in your Dock so don’t try to kill it or remove it while Comskip is identifying commercials in a MPEG file.

Also, I’ve configured Comskip to automatically start up and begin scanning for commercials on-the-fly as EyeTV3 is recording the TV Show. This is nice, as you don’t need to wait several minutes after the show has been completed to begin the commercial scanning. The scanning occurs while the show is being recorded. To make this configuration, you need to change the comskip.ini file (located in the directory that the comskip.exe file is found) and make these setting changes:

live_tv=1
live_tv_retries=16

Also, ETVComskip uses a freeware version of Comskip which works ok with most MPEG video files, but doesn’t work with HD recorded files. Since the EyeTV HD encoder that I’m using records in h.264 HD format, the freeware version of Comskip won’t work. I needed to get the “donor” version of Comskip from the developer which can successfully scan for commercials in the h.264 HD MPEG files and run faster than the freeware version. To get this donor version, I needed to donate a small amount of money to the developer (in my case, I donated $10) and then the developer gave me access to the special donor version. I then extracted all the files for the donor version of Comskip and replaced the same files in the ETVComskip folder with them.

Ok, that was how I setup my system for recording TV shows from a Comcast set top box with my iMac using EyeTV HD, with the ETVComskip utility for identifying commercials. The next step, is developing a method for using the identified commercial segment blocks from the Comskip analysis and cutting the commercials out of the MPEG recording. Here’s what I needed to do:

  1. Use the identified commercial segment blocks (stored in an .edl file) from Comskip and cut out the commericals from the MPEG recording file.
  2. Add meta data from the TV Show (e.g., show title, category, episode number, etc) to the edited MPEG file.
  3. Copy the new MPEG file to a special folder so that iTunes will sync the file to your iPod Touch or iPhone.
  4. Clean up by deleting any temporary files from this process.

So that’s the basic process. Below is the current Applescript (RecordingDone.scpt) that I’m using to post-process my EyeTV recorded shows. I’ve heavily commented the script to describe what each line of code is doing:

Now, aside from all the tricky scripting to extract the meta data like the TV show title, episode name, season and episode number, etc. there are three main sections that do the heavy lifting.

First, I have to comment that EyeTV3 will store two different video and audio streams in the recorded MPEG file. It stores one video stream for 1920×1080 resolution and another for 480×360. In my case, I wanted the lower-resolution video stream since ultimately I want my edited video for playback on my iPod Touch. So, I needed to use a freeware tool called ffmpeg to extract the video and audio stream data for the 480×360 resolution and write it to a temporary MPEG file. In my script, you’ll see that I’m using the arguments “-map 0:2   -map 0:3” for the call to ffmpeg which does what I just described above. It uses the same video and audio settings as the source file, so all I’m doing is extract the 2nd video/audio stream. If you want to process a recorded video for an iPad or other higher-resolution device, you might want to use the first video/audio stream and use “-map 0:0  -map 0:1” instead.

The next step is to use the freeware utility called mencoder which will remove the commercials from the temp MPEG file I created in in the previous step. Mencoder will take the .edl file generated by Comskip and create a new MPEG file cutting out the specified commercial segments. From the script you’ll see that there’s lots of arguments associated with mencoder that are required to get the proper audio format (the video format is copied over as-is from the source file). I also have the argument “-lavf  -lavfopts  format=ipod” which has mencoder set some atom tags in the created MPEG file so that iTunes will identify the video file as iTunes-compatible. If you don’t have this option, iTunes won’t recognize your video file and you can’t sync it. Finally, I have this argument defined “-mc 0 -ss 10” which cuts out the first 10 seconds of the recorded video (I do this, because EyeTV3 records video when it changes the channel to the desired recorded TV show and I want that stuff cut out).

So at this point we have an edited MPEG file with all the commercials cut out (yea!) and the atom tag is set for recognition by the iTunes software for syncing with your iPod Touch. Now, if I don’t do anything further iTunes will identify this video file as a “Movie” and will sync it as such. Since I want this recording to be identified as a “TV Show” I need to add in some specific meta data that iTunes can use for identification. To do this, I use a freeware tool called atomicparsley which will allow me to set the required atom tags.

So you’ll see in my script that I use atomicparsley to define the TV show title, episode name and description, season and episode number, etc. Once this is all added, the final MPEG file is ready for syncing to iTunes with all the required data for perfect identification.

So, how do we get iTunes to sync this edited video file? The easiest way to do this is to move this video file into a special iTunes folder after which the iTunes software will move it to the proper sub-folder and sync it to your iPod Touch. On my machine this folder location is, “/Users/dave/Music/iTunes/iTunes Media/Automatically Add to iTunes” on your machine you should have something similar. So I use a script command to move the edited video file to this folder.

Finally, I delete the temporary MPEG file that I created earlier and write a simple line of text to a log file. And that’s it. All the rest of the lines of coding in the script was added by the ETVComskip installer.

Now, this all gets processed automatically every time a show is recorded. As such, I don’t have to lift a finger to have all the commercials cut from my TV show recordings. So far, Comskip has done a good job identifying just the commercials and I’m not losing any good show content.

So you might ask, why do I go through all the trouble of writing this Applescript? Why not skip past the commercials manually as I’m watching them on my iPod Touch? For me, it’s actually a real pain to skip past the commercials since the video player on my iPod Touch has a very sensitive slider bar for positioning control. I can spend several seconds sliding back and forth trying to not miss the beginning of a show segment. With all the commercials removed, I don’t have to deal with any of that. Also, the final edited video file is significantly smaller than having one with the included commercials so that is a big plus.

For those who want to go down this path, here are some good resources:

Elgato EyeTV Forums – There are some good discussion threads on commercial detection and removal for the Mac system.

ffmpeg and mencoder tools – Here’s a good link for those who want to download and compile the ffmpeg and mencoder tools for their Mac (nice step-by-step instructions).

Atomic Parsley – This is the main link for getting Atomic Parsley working on your Mac system.

Comskip – This is the main site with a very good user forum that has details on fine tuning Comskip for better commercial detection (and where you can make a small donation to get the faster donator version of Comskip).

Advertisements

13 Responses to TV show commercial cutting – Part 2

  1. Jill says:

    Thank you for the detailed explanation. I’ve downloaded the donor and non donor versions of Comskip and ETVcomskip and I can’t get the non donor version to install correctly. There are no folders or files being created at

    /Library/Application Support/ETVComskip,
    and /Library/Application Support/EyeTV/Scripts/TriggeredScripts

    I’ve tried to redownload and reboot, but I think I’ll have to manually move them. Can you list which files go in each of those folders so I can copy them manually?

    • zunetips says:

      This is what I did to install the donator version of Comskip:

      First, I installed the freeware version of ETVComskip following the instructions provided by its makers. Next, I downloaded the latest donator version of Comskip from the web site provided by it’s maker from the email they sent me. I unzipped the downloaded file which appeared as a folder called “comskip80_039_donators” that contained several files.

      Next, I went to the folder “Macintosh HD/Library/Application Support/ETVComskip” and renamed the folder called “comskip” to “comskip_old”. Then, I copied the folder “comskip80_039_donators” over to the “Macintosh HD/Library/Application Support/ETVComskip” folder and renamed it to “comskip”. So effectively I replaced the “comskip” folder that was installed by the ETVComskip installer with the donator’s version.

      That should now allow the system to use the donator version of Comskip to process the recorded files. Hopefully, that will help get your setup working, but remember there’s lots of other places where something could still go wrong with the setup! 🙂

  2. Ian says:

    Hello. Thank you so much for putting this together. It is exactly what I need! Only problem is I am getting an error when I try to manually run the AppleScript and I cannot figure out what is wrong. 😦

    When manually run, it errors out at the first “do shell script cmd3”. It looks like it has to do with ffmpeg but I am not sure. When you made ffmpeg, what configure commands did you use? I used the options from the link on your page…

    ./configure –cc=gcc-4.2 –enable-gpl –enable-libfaac –enable-pthreads –enable-ffplay –enable-nonfree –enable-libx264

    Is that what you used? Also, the link to make ffmpeg talks about installing quite a number of other applications other then ffmpeg and MEncoder. Did you install those other applications also (like SoX, X264, Yasm, faac, etc…)

    Sorry for the newbie questions. 🙂

    Thanks!

    • zunetips says:

      It’s been a really long time since I worked with these Applescripts and this commercial removing process, so I don’t seem to remember the details to clearly. For the FFMPEG executable, I basically followed the instructions on this page: https://sites.google.com/site/llee040/Home/hdtv_to_dvd/

      • Ian says:

        Yeah, that is what I was afraid of. Looks like that page was updated last month and a lot of the page was changed to support what they were doing.

        Thanks anyway though, your script has really got me going in the right direction. Thanks!

      • Ian says:

        Sorry, one more request. Could you run a “ffmpeg” from a terminal window and post that? It should tell what configuration you used.

        Thanks!

  3. zunetips says:

    Here you go:

    FFmpeg version SVN-r25605, Copyright (c) 2000-2010 the FFmpeg developers
    built on Oct 30 2010 14:18:49 with gcc 4.2.1 (Apple Inc. build 5664)
    configuration: –arch=x86_64 –disable-ffserver –enable-gpl –enable-libfaac –enable-pthreads –enable-ffplay –enable-nonfree –enable-libx264
    libavutil 50.32. 3 / 50.32. 3
    libavcore 0. 9. 1 / 0. 9. 1
    libavcodec 52.93. 0 / 52.93. 0
    libavformat 52.84. 0 / 52.84. 0
    libavdevice 52. 2. 2 / 52. 2. 2
    libavfilter 1.53. 0 / 1.53. 0
    libswscale 0.12. 0 / 0.12. 0
    Hyper fast Audio and Video encoder
    usage: ffmpeg [options] [[infile options] -i infile]… {[outfile options] outfile}…

    • Ian says:

      Thanks! I appreciate it.

    • Ian says:

      Thanks again for running that command for me. After re-configuring ffmpeg using your options (rather then the options from the linked webpage) everything worked as expected. I ran with the following…

      ./configure –cc=gcc-4.2 –arch=x86_64 –disable-ffserver –enable-gpl –enable-libfaac –enable-pthreads –enable-ffplay –enable-nonfree –enable-libx264

    • zunetips says:

      I’m glad you got ffmpeg working on your Mac. Good luck with your commercial cutting efforts!

  4. Bill says:

    Thanks for such a great run through on how to do this! I have hobbled together my own version that does something similar but using handbrake instead. The piece I am/was missing is the season and episode information.

    Two questions:
    1) Do you know if there is a significant speed difference between using mencoder versus handbrake?
    2) Can you send me your completed script? The screen shot post cuts of some of the key shell commands I am looking for to get the season and episode info.

    Thanks again!

    Cheers,
    Bill

    • zunetips says:

      First, I haven’t used this commercial cutting procedure in a long time, and I’m not just leaving the commercials in my recorded videos. I was having issues with good scenes getting cut out, commercials being left in, and sometimes audio-video sync problems. So, I decided to just leave the commercials in and skip them manually so I don’t miss any important scenes in my show. At some point, I’ll probably go back and investigate this commercial cutting procedure again.

      Having said that, I don’t remember if I did a speed comparsion between mencoder and handbrake. If I had to guess, I would say handbrake would be faster but I don’t know for sure.

      I’ll see if I can dig up the script you are asking for and will try to make it available.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: