Can’t Remove the Commercials!!!!

July 29, 2012

If you search for “comskip” among my older postings, you’ll see that at one time I had developed a procedure for identifying and cutting out commercials from my video recordings. I used an application called Comskip (which is a Windows app so I ran it using the Wine cross-platform emulator) for identifying commercials in a video, ffmpeg and mencoder for video data manipulation and cutting. Unfortunately, with the upgrade in my video recording software (EyeTV3) these scripts and procedures no longer work.

I’ve spent hours and long evenings trying to develop a new method, but I’ve hit a big roadblock. I can still use Comskip to identify the commercial time points in a video file, but I have no way of automatically editing the video to remove the commercial segments. In the past, I used mencoder to do the video cutting in batch mode, but now it just won’t work with the MPEG-2 file that is created by the recording software. I tried extracting the video from the original MPEG-2 file first and using that in mencoder, but no dice. I can remove the commercials  manually using the EyeTV3 software, but I want an automated procedure that just does it for me.

Unfortunately, other video processing programs such as ffmeg and Handbrake can’t cut out the commercials from a video file as effectively as how mencoder did it. So, I’m stuck now with lots of commercials that I have to manually skip by. It’s not a super big deal, but when you’re use to watching a show with no commericals, it’s hard to go back!

So if anyone reading this blog posting has any ideas or suggestions, please let me know! 🙂

Updated (12 Aug 2012): Ok, I finally figured out how to do this. I won’t go into the nitty gritty details, but the following Unix command on my iMac system is what I use to identify the commercials in a EyeTV recorded raw mpeg-2 file then create a new mp4 file with the recordings stripped out:

export DISPLAY=:0.0; /usr/bin/nice -n 5 “/Library/Application Support/ETVComskip/Wine.app/Contents/Resources/bin/wine” “/Library/Application Support/ETVComskip/comskip/comskip.exe” –ini=”/Library/Application Support/ETVComskip/comskip/comskip.ini” /Users/dave/Documents/EyeTV\ Archive/Enemy\ of\ the\ State.eyetv/0000000015cdc458.mpg &> /dev/null; /bin/sleep 10; /usr/local/bin/mencoder /Users/dave/Music/iTunes/iTunes\ Media/Android\ Videos/Enemy\ of\ the\ State.mp4 -o /Users/dave/Music/iTunes/iTunes\ Media/Android\ Videos/Enemy\ of\ the\ State\ \(No\ Commercials\).mp4 -edl /Users/dave/Documents/EyeTV\ Archive/Enemy\ of\ the\ State.eyetv/0000000015cdc458.edl -ovc copy -oac faac -faacopts mpeg=4:object=2:raw:br=164 -srate 48000 -of lavf > /dev/null 2>&1 &

I basically use Comskip to identify the commercials in the original mpeg-2 video file, then use Mencoder to strip out the commercials from the converted video (which was created using HandbrakeCLI, converting the original mpeg-2 file to mp4 format) and create a new mp4 file. The trick, was getting the latest version of Mencoder downloaded, compiled, and installed on my iMac system.


Watching live TV on my iPhone 4

April 17, 2011

At home, I’ve got an EyeTV HD device connected to a Comcast digital cable box for recording TV shows on my iMac desktop computer. The system is entirely automated, where scheduled shows are recorded, commercial removed, and finally converted to iPhone format and placed in iTunes for syncing. This allows me to watch my favorite shows at my leisure, usually at night on the couch or in bed. A few weeks ago I was on a business trip and I forgot to schedule my EyeTV to record an episode of Stargate: Universe, and it was not playing on any local station in my hotel room. What was I suppose to do?

Luckily, I have an app on my iPhone that allows me to control the EyeTV HD software on my iMac at home and also watch live TV through it via the Internet. So, I decided to give it a try, even though the hotel’s WiFi connect speed was incredible slow and virtually unusable. So instead of trying to use WiFi, I decided to just use my cellular connection via Verizon to stream the Live TV from my iMac at home to my iPhone.

Surprisingly, the video stream worked great! I was able to watch my show with no pausing or dropping of the TV streaming image. On a few occasions the image got a bit grainy (heavy pixels), but for the most part it looked just like watching an mp4 video file on my iPhone.

If I do switch to an Android phone in the near future, this will be one thing that I’ll miss. I don’t use this feature a lot, but when I do it works great.


Stargate Universe – CANCELLED

December 31, 2010

Looks like the SyFy channel decided to pull the plug on the latest Stargate Universe series. The producers of this incarnation tried to change the format a bit by making it edgier and darker than the previous Stargate Series, but it was a really slow moving show and I think that coupled with franchise-fatigue really killed it. I watched it weekly because I’m a big SciFi fan and it had some interesting moments, however, with these types of franchise series you see repeating themes with the shows that does get old and tiresome.

Now, if MGM came out with some good Stargate DVD movies (either SG-1, Atlantis, or Universe) I would certainly purchase it. But for now, I think Stargate has suffered the same fate as Star Trek and needs some rest before MGM tries to conjure up a new Stargate series.


Final Capria episodes airing Tuesday

December 31, 2010

In all their great wisdom, the SyFy (formerly Sci-Fi) Channel decided to cancel the Battlestar Galactica Series, Caprica mid-way through the 2nd season. If you want to watch the final 5 unaired episodes, get ready for the Caprica Marathon on Tuesday, January 4th. All 5 episodes will be shown back-to-back starting at 7 pm PST. So if you want to find out what happens to the Graystones, Adamas, and the birth of the Cylon destroyers, dial in on Tuesday for your last chance!

UPDATE: I’ve recorded and watched the first 4 new episodes of Caprica, and I understand why SyFy cancelled the show. All of the episodes were really, really slow with very little real action. Since I recorded them, I could easily fast-forward through the boring and slow parts, but even then the story was a bit long in the tooth. I pretty much fast-forwarded through all the scenes with Amanda Graystone, and most of the Zoey scenes (even though Zoey is a key component in the Cylon mythos). In my opinion, the real interest is with the birth of the Cylons and what caused them to war with their creators, and if the writers focused on that instead of all the character background stories it would have had a better chance of continuing on….


Comskip fine tuning

November 7, 2010

As mentioned in my previous posting, I’m using several freeware utilities on my Mac to identify and remove commercial segments from recorded TV shows. Comskip is the tool I use to identify the commercials, and it has numerous setting configurations that you can change to help fine tune the commercial identification process.

Now, these settings can vary based on where you made the recording. For example, settings for USA broadcasts may not be optimal for European broadcasts. As such, fine tuning is required if you want Comskip to catch all (or most) of the commercials without cutting out good TV show content.

There several postings regarding tuning on the Comskip forums which I’ve read over. The basic process is trial-and-error, making logical changes to the Comskip settings file and examining the results. It is a lengthy process, but well worth it if you want to get Comskip working at peak performance.

In my case, I spent a few hours getting Comskip tuned for two typical programs that I record. One is Fringe, and the other is Design to Sell. Fringe is a good example of a typical mainstream network TV show that I like to watch, and Design to Sell is a typical HGTV home improvement show.

For fine tuning, you need to change some set values in a file called comskip.ini (located in the same folder as the comskip.exe program). I changed the following three lines:

detection_method = 255

Adjusting the detection method to 255 has Comskip use all of the various commercial detection methods.

verbose = 1

Changing the verbose setting from the default 10 to 1 reduces the amount of analysis information written to the Comskip log file for the analyzed video. I could have changed this value to zero (to not write any data), but I left it at 1 so I can try to figure out why Comskip identifies certain block segments as a commercial vrs show content.

max_volume=50

The max volume setting defines whether Comskip marks a segment block as commercial or show content by the volume level. If the suspected commercial segment has a volume level higher than the max_volume setting, then Comskip marks it as show content. The idea is, when a show transitions to or from a commercial you may have a brief black-screen with very little volume being played (this will mark a commercial segment). The default value of 500 works ok for my system, but it didn’t catch the commercial segments quite right. Since I’m recording a digital signal, the creator of Comskip recommended using a value of 50 since the audio signal is clearer.

So with these settings the commercial detection for the Fringe show is very, very good. There’s only one spot in the show where the Comskip misses a commercial and that’s because the TV Network changed the location and style of the broadcast “logo” on the screen (why, I don’t know). These same settings did a very good job with the Design to Sell program also (but not perfect).

So for now I’m going to stick with these settings and see how well they work out. If they miss a few seconds of show content I’m ok with that, however, if Comskip misses several minutes of good show content I’ll have to do some more fine tuning.

Other settings that you can change are:

max_brightness
test_brightness
max_avg_brightness

If Comskip is not identifying commercials in your show, you can increase the value of these variables (by increments of 1) and see how much improvement you can get.

Another cool feature that Comskip has is the ability to look for defined words in the Closed Caption (CC) text for a TV show. You can define individual words in a file called comskip.dictionary (found in the Comskip install folder) . The words above the “——-” line identify good TV content while words below that line identify a commercial segment. I think this is a brilliant method for helping Comskip identify commercials, however, I unfortunately can’t make use of it. The reason being, I can’t seem to get EyeTV3 to store the CC text in the MPEG file. It’s either an issue with the EyeTV3 software or with my Comcast digital set top box.

Update: So far, Comskip has been able to identify the correct commercial segments using theses settings for the following TV Shows:

Fringe, Design to Sell, The Walking Dead (AMC), Stargate SG-1

What nice, is that all of these shows have a slightly different broadcast and format, yet Comskip is able to catch the commercials quite effectively. I’m actually very pleased, as I was expecting to have fiddle with the settings a bit more to accomplish this accuracy. Thanks to the author of Comskip!


TV show commercial cutting – Part 2

November 7, 2010

I’ve had a few people ask about my Applescript which I use to post-process TV show recordings created by my Elgato EyeTV HD encoder, so here’s some more details.

First, I have this basic configuration:

  1. Comcast digital signal fed into a Comcast RG110 digital set top box
  2. Video/Audio component output from the Comcast set top box are fed into an Elgato EyeTV HD encoder
  3. EyeTV HD encoder is connected to my Apple iMac desktop computer via a USB 2.0 cable
  4. The EyeTV3 software on the iMac controlling the recording of scheduled TV shows
  5. IR Changer (Blaster) connected to the Comcast set top box allowing the EyeTV HD encoder to change the TV stations appropriately

I also have installed the freeware tool called ETVComskip which is used to identify commercial segments in a recorded MPEG file and saves them to a text file (with an .edl extension). This is the primary tool that I use for commercial detection, so I’ve spent a lot of time tuning the settings for this application to get good commercial detection (which I’ll describe in a future posting).

Luckily, the EyeTV3 software is able to run two specific Applescripts when a recording has just started (RecordingStarted.scpt) and when it has finished (RecordingDone.scpt). When you install the ETVComskip application it will automatically modify both of these scripts to support the commercial detection process.

Now, the ETVComskip tool is basically a Python script that uses a program called Comskip to detect the commercials in an MPEG video file. The Comskip app is actually a Microsoft Windows application, so the Python script is setup to use Wine (WIN32 emulator) to run the Comskip windows executable on your Mac. You don’t need to worry about all of this, as the makers of ETVComskip have taken care of everything. The only thing to note, is that you will see the X-Windows App icon appear in your Dock so don’t try to kill it or remove it while Comskip is identifying commercials in a MPEG file. Read the rest of this entry »


Automating commercial cutting from recorded TV shows

October 31, 2010

Back when I was using Microsoft Media Center to record TV shows on my Windows 7 PC (with cable set top box and TV Tuner card) I also used a nifty freeware tool called DVRMSToolbox (dtb) to help cut out commercials from my recordings and convert the videos to a format acceptable for my Microsoft Zune HD. Since switching to an Apple iMac desktop system, I had to leave my Microsoft Zune HD for an Apple iPod Touch (which I got last Friday for my birthday). In fact, if you read some of my previous postings you’ll see that I’ve gone to an all-Mac system with regards to recording TV shows using my iMac.

Consequently, I don’t have the ability to automatically remove commercials from my TV recordings (since DVRMSToolbox only runs on Windows systems). After lots of Googling, I came up with a system that might just do the trick for removing commercials from my TV shows and make them acceptable for iTunes syncing with my iPod Touch. Here’s the procedure that I’ve implemented:

  1. Using the EyeTV HD hardware with the EyeTV3 software, I’ve set up scheduled TV show recordings on my iMac. (The system records shows in MPEG4 H264 file format)
  2. While the show is recording, I’m using a program called Comskip that will on-the-fly identify the timing segments of all commercials. This procedure is initiated by EyeTV3 automatically by running an AppleScript when the show begins recording.
  3. After the recording is completed for a show, EyeTV3 will call another AppleScript that allows for post-processing. I do the following after each show is finished recording:
    1. Wait 60 seconds for the Comskip program to finish up its analysis of the show.
    2. Get the show’s metadata (e.g., title, description, etc.) and save it
    3. Run a program called ffmpeg that will extract the proper video and audio stream from the raw recorded MPEG4 file.
    4. Run a program called mencoder that will take the commercial break information from Comskip and cut out all the commercials.
    5. Run a program called atomicparsley that will insert the TV show’s metadata into the new mp4 video file.
    6. Copy this commercial-free video file of the recorded TV show to the iTunes folder called “Automatically Add to iTunes” which will automatically process the file and make it available for syncing with my iPod Touch.
    7. Do some housecleaning by deleting temp files, etc. and also note in a log file that this recording has been processed.

Thankfully,  the makers of EyeTV3 (Elgato) have added provisions to fire off certain AppleScripts so I can do this post-processing. I’ve spend all weekend downloading and compiling source code for programs such as ffmpeg, mencoder, and atomicparsley and doing some heavy AppleScript writing. Note, that I’ve never written AppleScripts before, and it sure does help to have lots of examples available on the Internet!

So, it seems that my test cases are working ok with this process, so I’ll have to let it run for a few days to make sure everything is handled properly. I’m expecting that I’ll need to tweak the Comskip settings file to help it identify commercials better, but that’s for another day!

One thing to note: On my Zune HD I had a skip forward/backward button that allowed me to skip ahead several seconds to pass up commercials if needed. Unfortunately, the iPod Touch does not have such a feature and my only recourse is to carefully use the slider bar control to skip commercials. Since this slider control is very sensitive, it takes a few attempts to pass a commercial without missing too much of my show. Thus, I decided to go down the path of auto editing the commercials to avoid this issue.

Updated: Here’s a good link that describes how to download and compile the ffmpeg and mencoder programs for your Mac: http://lonestar.utsa.edu/llee/HDTV_to_DVD.html

Also, I’ve uploaded my recordingDone.scpt file to the Cloud in case you EyeTV users want to examine it for your use. You can download it from this link.