Comskip fine tuning

November 7, 2010

As mentioned in my previous posting, I’m using several freeware utilities on my Mac to identify and remove commercial segments from recorded TV shows. Comskip is the tool I use to identify the commercials, and it has numerous setting configurations that you can change to help fine tune the commercial identification process.

Now, these settings can vary based on where you made the recording. For example, settings for USA broadcasts may not be optimal for European broadcasts. As such, fine tuning is required if you want Comskip to catch all (or most) of the commercials without cutting out good TV show content.

There several postings regarding tuning on the Comskip forums which I’ve read over. The basic process is trial-and-error, making logical changes to the Comskip settings file and examining the results. It is a lengthy process, but well worth it if you want to get Comskip working at peak performance.

In my case, I spent a few hours getting Comskip tuned for two typical programs that I record. One is Fringe, and the other is Design to Sell. Fringe is a good example of a typical mainstream network TV show that I like to watch, and Design to Sell is a typical HGTV home improvement show.

For fine tuning, you need to change some set values in a file called comskip.ini (located in the same folder as the comskip.exe program). I changed the following three lines:

detection_method = 255

Adjusting the detection method to 255 has Comskip use all of the various commercial detection methods.

verbose = 1

Changing the verbose setting from the default 10 to 1 reduces the amount of analysis information written to the Comskip log file for the analyzed video. I could have changed this value to zero (to not write any data), but I left it at 1 so I can try to figure out why Comskip identifies certain block segments as a commercial vrs show content.

max_volume=50

The max volume setting defines whether Comskip marks a segment block as commercial or show content by the volume level. If the suspected commercial segment has a volume level higher than the max_volume setting, then Comskip marks it as show content. The idea is, when a show transitions to or from a commercial you may have a brief black-screen with very little volume being played (this will mark a commercial segment). The default value of 500 works ok for my system, but it didn’t catch the commercial segments quite right. Since I’m recording a digital signal, the creator of Comskip recommended using a value of 50 since the audio signal is clearer.

So with these settings the commercial detection for the Fringe show is very, very good. There’s only one spot in the show where the Comskip misses a commercial and that’s because the TV Network changed the location and style of the broadcast “logo” on the screen (why, I don’t know). These same settings did a very good job with the Design to Sell program also (but not perfect).

So for now I’m going to stick with these settings and see how well they work out. If they miss a few seconds of show content I’m ok with that, however, if Comskip misses several minutes of good show content I’ll have to do some more fine tuning.

Other settings that you can change are:

max_brightness
test_brightness
max_avg_brightness

If Comskip is not identifying commercials in your show, you can increase the value of these variables (by increments of 1) and see how much improvement you can get.

Another cool feature that Comskip has is the ability to look for defined words in the Closed Caption (CC) text for a TV show. You can define individual words in a file called comskip.dictionary (found in the Comskip install folder) . The words above the “——-” line identify good TV content while words below that line identify a commercial segment. I think this is a brilliant method for helping Comskip identify commercials, however, I unfortunately can’t make use of it. The reason being, I can’t seem to get EyeTV3 to store the CC text in the MPEG file. It’s either an issue with the EyeTV3 software or with my Comcast digital set top box.

Update: So far, Comskip has been able to identify the correct commercial segments using theses settings for the following TV Shows:

Fringe, Design to Sell, The Walking Dead (AMC), Stargate SG-1

What nice, is that all of these shows have a slightly different broadcast and format, yet Comskip is able to catch the commercials quite effectively. I’m actually very pleased, as I was expecting to have fiddle with the settings a bit more to accomplish this accuracy. Thanks to the author of Comskip!


TV show commercial cutting – Part 2

November 7, 2010

I’ve had a few people ask about my Applescript which I use to post-process TV show recordings created by my Elgato EyeTV HD encoder, so here’s some more details.

First, I have this basic configuration:

  1. Comcast digital signal fed into a Comcast RG110 digital set top box
  2. Video/Audio component output from the Comcast set top box are fed into an Elgato EyeTV HD encoder
  3. EyeTV HD encoder is connected to my Apple iMac desktop computer via a USB 2.0 cable
  4. The EyeTV3 software on the iMac controlling the recording of scheduled TV shows
  5. IR Changer (Blaster) connected to the Comcast set top box allowing the EyeTV HD encoder to change the TV stations appropriately

I also have installed the freeware tool called ETVComskip which is used to identify commercial segments in a recorded MPEG file and saves them to a text file (with an .edl extension). This is the primary tool that I use for commercial detection, so I’ve spent a lot of time tuning the settings for this application to get good commercial detection (which I’ll describe in a future posting).

Luckily, the EyeTV3 software is able to run two specific Applescripts when a recording has just started (RecordingStarted.scpt) and when it has finished (RecordingDone.scpt). When you install the ETVComskip application it will automatically modify both of these scripts to support the commercial detection process.

Now, the ETVComskip tool is basically a Python script that uses a program called Comskip to detect the commercials in an MPEG video file. The Comskip app is actually a Microsoft Windows application, so the Python script is setup to use Wine (WIN32 emulator) to run the Comskip windows executable on your Mac. You don’t need to worry about all of this, as the makers of ETVComskip have taken care of everything. The only thing to note, is that you will see the X-Windows App icon appear in your Dock so don’t try to kill it or remove it while Comskip is identifying commercials in a MPEG file. Read the rest of this entry »


I Love my Mac!

November 2, 2010

I’ve had my 27″ iMac desktop for a few months now, and I really love the fact that the Mac OS X is Unix under the hood. I’m an old Unix user from the 1990’s and it’s great that I can pop open a terminal shell window on my Mac and type in standard Unix/Linux commands to do my dirty work. For example, I had some video files in AVI format that I wanted converted to MP4 format for my iPod Touch, so I downloaded and compiled GPL source code for Mencoder and AtomicParsley on my Mac. I then typed in this command line line entry in a terminal shell window to do the conversion process:

mencoder stargate.avi -ovc copy -oac copy stargate.mp4

And off went the conversion process. I then used AtomicParsley to add in metadata information so that it would appear in iTunes properly:

atomicparsley stargate.mp4 –stik “Movie” –title “Stargate” –artwork sg.jpg –overWrite

I then copied the stargate.mp4 file over to the “Add to Itunes Automatically” folder to have the movie file placed in the proper directory for iTunes syncing with my iPod Touch. Sweet!

In this case, an alternative to using mencoder in a terminal shell window is to use the freeware product called Handbrake which has a GUI and works really well for converting videos. I tested it with a few AVI files and they converted over flawlessly to MP4 format for my iPod Touch.


Automating commercial cutting from recorded TV shows

October 31, 2010

Back when I was using Microsoft Media Center to record TV shows on my Windows 7 PC (with cable set top box and TV Tuner card) I also used a nifty freeware tool called DVRMSToolbox (dtb) to help cut out commercials from my recordings and convert the videos to a format acceptable for my Microsoft Zune HD. Since switching to an Apple iMac desktop system, I had to leave my Microsoft Zune HD for an Apple iPod Touch (which I got last Friday for my birthday). In fact, if you read some of my previous postings you’ll see that I’ve gone to an all-Mac system with regards to recording TV shows using my iMac.

Consequently, I don’t have the ability to automatically remove commercials from my TV recordings (since DVRMSToolbox only runs on Windows systems). After lots of Googling, I came up with a system that might just do the trick for removing commercials from my TV shows and make them acceptable for iTunes syncing with my iPod Touch. Here’s the procedure that I’ve implemented:

  1. Using the EyeTV HD hardware with the EyeTV3 software, I’ve set up scheduled TV show recordings on my iMac. (The system records shows in MPEG4 H264 file format)
  2. While the show is recording, I’m using a program called Comskip that will on-the-fly identify the timing segments of all commercials. This procedure is initiated by EyeTV3 automatically by running an AppleScript when the show begins recording.
  3. After the recording is completed for a show, EyeTV3 will call another AppleScript that allows for post-processing. I do the following after each show is finished recording:
    1. Wait 60 seconds for the Comskip program to finish up its analysis of the show.
    2. Get the show’s metadata (e.g., title, description, etc.) and save it
    3. Run a program called ffmpeg that will extract the proper video and audio stream from the raw recorded MPEG4 file.
    4. Run a program called mencoder that will take the commercial break information from Comskip and cut out all the commercials.
    5. Run a program called atomicparsley that will insert the TV show’s metadata into the new mp4 video file.
    6. Copy this commercial-free video file of the recorded TV show to the iTunes folder called “Automatically Add to iTunes” which will automatically process the file and make it available for syncing with my iPod Touch.
    7. Do some housecleaning by deleting temp files, etc. and also note in a log file that this recording has been processed.

Thankfully,  the makers of EyeTV3 (Elgato) have added provisions to fire off certain AppleScripts so I can do this post-processing. I’ve spend all weekend downloading and compiling source code for programs such as ffmpeg, mencoder, and atomicparsley and doing some heavy AppleScript writing. Note, that I’ve never written AppleScripts before, and it sure does help to have lots of examples available on the Internet!

So, it seems that my test cases are working ok with this process, so I’ll have to let it run for a few days to make sure everything is handled properly. I’m expecting that I’ll need to tweak the Comskip settings file to help it identify commercials better, but that’s for another day!

One thing to note: On my Zune HD I had a skip forward/backward button that allowed me to skip ahead several seconds to pass up commercials if needed. Unfortunately, the iPod Touch does not have such a feature and my only recourse is to carefully use the slider bar control to skip commercials. Since this slider control is very sensitive, it takes a few attempts to pass a commercial without missing too much of my show. Thus, I decided to go down the path of auto editing the commercials to avoid this issue.

Updated: Here’s a good link that describes how to download and compile the ffmpeg and mencoder programs for your Mac: http://lonestar.utsa.edu/llee/HDTV_to_DVD.html

Also, I’ve uploaded my recordingDone.scpt file to the Cloud in case you EyeTV users want to examine it for your use. You can download it from this link.


Recording TV shows on your Mac

September 14, 2010

For the last several years my wife and I have been using Microsoft’s Windows Media Center to schedule the recording of TV shows on our respective PC machines. In general WMC has been working well, with just a few glitches here and there. Using some external tools (i.e.,DVRMSToolbox and ShowAnalyzer) I have a process in place to record TV shows, cut out the commercials, and converted the recorded shows to WMV format for our Zune media players.

Last year, out of frustration my wife switched from her Vista PC to an Apple MacBook Pro laptop. Since she can’t run the Microsoft Zune syncing software on her MacBook Pro, I kept her PC still up and running to continue recording TV shows and processing them for her Zune 120 player. A few months ago her Zune device stopped working (hard drive failure) so for Mother’s Day I bought her an Apple iPod Touch (32 GB) as a replacement. Mainly because it will easily work with her MacBook Pro using the iTunes software. She still uses her Vista Desktop PC for recording TV shows, but now I’ve modified some of the processing scripts to convert the recordings to MP4 format for her iPod Touch.

Unfortunately, her Vista PC was having trouble recording her favorite TV shows for some reason or another (e.g., couldn’t download the latest TV listings, conversions stalled because of Windows updates, etc.). Twice, I had to reinstall the WMC software because the PC couldn’t download the TV listings from the Internet. So yesterday, I decided I would switch her over to a Mac-based system for recording TV shows from our cable TV service.

Upon doing some investigating, it seems that the product called EyeTV HD was what we needed. This relatively new product is designed to work specifically for the Mac OS X, and has the ability to work with cable set top boxes for changing channels via an IR Blaster. The only issue we currently have, is that this device requires component input for the video and audio, and our current set top box from Comcast only output a coaxial line.

So, I took our current converter box down to the local Comcast office and asked to switch it for a converter that has component output. Unfortunately for me, I didn’t specify I wanted HD component video output so they gave me an old Motorola converter box that has the very old composite video output (think of the old Pong game console that used the composite video connection) which just didn’t work for my needs. I did ask two different people in the Comcast office if this converter would give me digital component output and each of them said, “definitely” without even looking at the connectors on the back of the converter. As such, I came home to verify that this box wouldn’t work (as I suspected) and I called Comcast on the phone to see if they could mail me the right unit. So in about a week I’ll have the proper converter box to continue with my TV recording setup.

The nice thing about the EyeTV HD is that it has a onboard MPEG encoder, so it won’t use the MacBook Pro’s CPU for the encoding. Also, the EyeTV’s software can convert recorded TV shows to both iPad and iPod Touch formats at the same time. So if my wife chooses, she can watch her TV shows on either her iPad or iPod Touch. As such, I probably won’t need to jump through very many hoops to get her TV shows converted to the proper format for her iPod Touch, compared to what I needed to do for my Zune with Windows 7 and Vista.

If this all works out ok, I’ll probably switch over myself to a similar system and then I can shutdown my Windows 7 PC permanently.


Comcast Switched to All Digital… OK so far

June 14, 2009

comcastYesterday was the big day when Comcast in my area (Seattle) switched over to all digital. In preparation, I obtained two Set Top Boxes (STB) from Comcast a few weeks ago for my wife’s and my computer TV Tuner cards, and everything seems to be working ok. One thing I did notice on the first day, is that some channels were coming in distorted with big pixels (like when you view a dirty DVD), but that seemed to have cleared up.

So, the big switch over to digital hasn’t affected us much (aside from having to integrate a STB into my PVR system).


My Windows Media Center Remote Arrived Today!

May 23, 2009

tv_setIt looks like my Windows Media Center Remote arrived earlier than expected, so I spent some time this afternoon hooking up my new set of hardware components. As you may have read in a previous posting, Comcast in the Seattle area will be switching  some of their “premium” channels from analog to digital signal. As such, I will need to use a Comcast Set Top Box (STB) which will convert these digital signals to analog for the TV Tuner Card inside my Desktop PC system. Unfortunately, I can’t control the STB directly using the Vista Media Center (VMC) software (which I use to schedule recordings, watch live TV, etc.) so I needed to purchase a Windows Media Center Remote (with Infrared Emitter) to do the channel changing.

Basically, I connect the IR receiver for the Windows Media Center Remote to my Desktop PC via a USB cable. Once I do that, the VMC software has the ability to interface with it. The IR receiver also has a jack for a IR emitter, which VMC will use to send IR signals to the STB to change its channel. So, I then connect the STB, with the Cable TV line connecting to it, and second connection between it and my TV Tuner card.

Once everything was connected up, I followed the installation guide on this web page. It was very helpful in explaining how to hook up the hardware and what the setup steps were with the VMC software. After following this guide, it appears that the entire setup is working as expected. No hangs or snags as of yet. Whew! 🙂

It seems so primitive to have to use IR emitters taped to the front of a STB, etc. to get all this to work, but I really have no choice since there isn’t a reasonable alterative. I’ll just have to keep an eye on the whole McGyvered system and will let you know if I have any issues. So for now, I’m back in business with recording TV shows on the Comcast Cable Network!

NOTE: If the audio volume for your recordings seem very low, try increasing the volume of the STB by using the remote control that came with it. I noticed that my recordings were really low in volume, and jacking up the STB volume resolved the issue.