Automating commercial cutting from recorded TV shows

October 31, 2010

Back when I was using Microsoft Media Center to record TV shows on my Windows 7 PC (with cable set top box and TV Tuner card) I also used a nifty freeware tool called DVRMSToolbox (dtb) to help cut out commercials from my recordings and convert the videos to a format acceptable for my Microsoft Zune HD. Since switching to an Apple iMac desktop system, I had to leave my Microsoft Zune HD for an Apple iPod Touch (which I got last Friday for my birthday). In fact, if you read some of my previous postings you’ll see that I’ve gone to an all-Mac system with regards to recording TV shows using my iMac.

Consequently, I don’t have the ability to automatically remove commercials from my TV recordings (since DVRMSToolbox only runs on Windows systems). After lots of Googling, I came up with a system that might just do the trick for removing commercials from my TV shows and make them acceptable for iTunes syncing with my iPod Touch. Here’s the procedure that I’ve implemented:

  1. Using the EyeTV HD hardware with the EyeTV3 software, I’ve set up scheduled TV show recordings on my iMac. (The system records shows in MPEG4 H264 file format)
  2. While the show is recording, I’m using a program called Comskip that will on-the-fly identify the timing segments of all commercials. This procedure is initiated by EyeTV3 automatically by running an AppleScript when the show begins recording.
  3. After the recording is completed for a show, EyeTV3 will call another AppleScript that allows for post-processing. I do the following after each show is finished recording:
    1. Wait 60 seconds for the Comskip program to finish up its analysis of the show.
    2. Get the show’s metadata (e.g., title, description, etc.) and save it
    3. Run a program called ffmpeg that will extract the proper video and audio stream from the raw recorded MPEG4 file.
    4. Run a program called mencoder that will take the commercial break information from Comskip and cut out all the commercials.
    5. Run a program called atomicparsley that will insert the TV show’s metadata into the new mp4 video file.
    6. Copy this commercial-free video file of the recorded TV show to the iTunes folder called “Automatically Add to iTunes” which will automatically process the file and make it available for syncing with my iPod Touch.
    7. Do some housecleaning by deleting temp files, etc. and also note in a log file that this recording has been processed.

Thankfully,  the makers of EyeTV3 (Elgato) have added provisions to fire off certain AppleScripts so I can do this post-processing. I’ve spend all weekend downloading and compiling source code for programs such as ffmpeg, mencoder, and atomicparsley and doing some heavy AppleScript writing. Note, that I’ve never written AppleScripts before, and it sure does help to have lots of examples available on the Internet!

So, it seems that my test cases are working ok with this process, so I’ll have to let it run for a few days to make sure everything is handled properly. I’m expecting that I’ll need to tweak the Comskip settings file to help it identify commercials better, but that’s for another day!

One thing to note: On my Zune HD I had a skip forward/backward button that allowed me to skip ahead several seconds to pass up commercials if needed. Unfortunately, the iPod Touch does not have such a feature and my only recourse is to carefully use the slider bar control to skip commercials. Since this slider control is very sensitive, it takes a few attempts to pass a commercial without missing too much of my show. Thus, I decided to go down the path of auto editing the commercials to avoid this issue.

Updated: Here’s a good link that describes how to download and compile the ffmpeg and mencoder programs for your Mac: http://lonestar.utsa.edu/llee/HDTV_to_DVD.html

Also, I’ve uploaded my recordingDone.scpt file to the Cloud in case you EyeTV users want to examine it for your use. You can download it from this link.


Dealing with two different calendars in life

October 28, 2010

For my day job I use Microsoft Outlook running on my Windows XP laptop for all my work-related appointments. Since my calendar is “shared” among my colleagues, I don’t put any personal appointments in it. Sure, I could always mark my personal appointments as “private” so only I can see the title and details, but I’m so paranoid that I would forget and thus have my private life displayed for all my co-workers to see.

Thus, I keep my personal calendar on my home machine (Apple iMac) using the provided iCal calendar application. The big challenge for me, is to be able to see both calendars on my home iMac, my work laptop, and on my Palm Pixi smartphone. So here’s what I did to make that happen:

First, my Palm Pixi can display multiple calendars provided by a single handful of sources. One is Microsoft Exchange and the other is Google Calendar. Since my work email through Outlook uses MS Exchange, that was a no brainer. So I have my work calendar on my Palm Pixi already figured out. Since my Palm Pixi can handle the Google Calendar, I needed to get my personal appointments synced or entered into Google Calendar in the cloud. I discovered the best way to do this is to subscribe to Google Calendar in Apple iCal and use it as my main personal calendar. I can make entries to Google Calendar in iCal and they will automatically be pushed up to Google Calendar in the Cloud. This way, my Palm Pixi can display both my personal (Google) calendar and work (Exchange) calendar.

With my personal calendar already in the cloud in Google Calendar, All I needed to do was subscribe to it in MS Outlook on my work laptop. Now I can view my personal appointments and work appointments on my work laptop using Outlook (albeit, I can only view my personal appointments, which is ok with me).

Finally, I needed to somehow get my work appointments from Outlook to be visible in Apple iCal on my Mac. I accomplished this, by publishing my Outlook appointments to a WebDav server in the cloud, specifically to a free file storage service called http://www.box.net. Basically, Outlook will periodically upload a single file (.ics) to the http://www.box.net file server and thus store my work appointments in the cloud in a simple text file. Next, I have Apple iCal subscribe to this WebDav server to download the calendar information in the .ics file every 15 minutes or so.

Now, I have a system that will allow me to view both my work and personal appointments on my home iMac, work laptop, and Palm Pixi smartphone without me having to copy and paste appointments between two separate calendars. Here’s what I can do for making changes:

1) My Palm Pixi can make updates to both my Personal and work calendars

2) Personal appointment changes are made on my home iMac.

3) Work appointment changes are made on my work laptop in Outlook.

I know that this all sounds like a pain, but I really needed to keep my personal and work appointments separate and have both viewable on different computers and devices.

Now, some might ask, “Why don’t you just have two separate calendars in Google Calendar and keep everything in one place, in the cloud?” That’s a good question. The biggest reason I’ve found, is that when I get a meeting invite in an email message, if I double-click on the attached .ics file my computer will try to open the .ics file using a stand-alone application on my system. So it will either try to use Apple iCal or Outlook or some other specified application. I don’t have the ability to have my web browser open and process the .ics file for entering it into Google Calendar. That just won’t work. Thus, I’m stuck with using an actual computer calendar application to easily process meeting invitations.

What I really would need, is a small application that can open the .ics file and place it in Google Calendar in the cloud. That, would be a really handy app!

 


Will Windows Phone 7 be a hit or dud?

October 26, 2010

The more I read online information about the new Windows Phone 7 devices, the more I’m hoping it will be a big success in the mobile phone industry. This new OS just seems very polished, fresh, and fun to use (based on the videos I’ve seen). I think most people want to try something different at times, and that may be a big driving factor for people to switch over to a Windows Phone 7 device after it is initially launched. The bad thing, is that it will appear on the AT&T and T-Mobile networks for the first several months before we start seeing the CDMA supported devices for Verizon and Sprint networks in the USA. That really sucks, since both AT&T and T-Mobile are known for having poor phone service and the highest rated dropped calls among all of the US cellular carriers.

I’m currently on the Sprint Network (which I love) and it would be a difficult to switch to a different carrier just to get a Windows Phone 7 device. I’d either have to keep my existing line with my Palm Pixi WebOS phone and get new service with AT&T (my choice over T-Mobile), or pay the early termination fees with Sprint to get out of my 2-year contract (which has about 1 more year to go). That wouldn’t be too bad, but going from a good cellular network to one that is known to drop calls would just suck big time. I travel throughout Washington State as well as several states in the Western half of the US, so I need reliable service in other areas than my home area.

Of course, I can always offset the cost of having and using an additional handset if I have a few apps bringing in money on the device’s online App Store (like I’m doing with my WebOS phone). If I could bring in just $200 per month that would definitely cover the cost of using a new handset on the AT&T network.

Most likely, I’ll take a wait-and-see approach and will keep an eye on how well the Phone 7 devices sell after November 8th. If they are a big hit and sell out everywhere, that would be a very good sign and I’ll have to really think hard about getting a phone. Also, once users begin using these phones in earnest on the AT&T/T-Mobile network we’ll probably see lots of reports of the good, bad, and ugly from real users.


Windows Phone 7 Apps… good or junk?

October 25, 2010

The new Microsoft Windows Phone 7 devices are starting to appear in Europe, and will be available early in November in the US. I just saw a blog posting stating that there are already over 700 apps available for the Phone 7 OS on the Zune Marketplace, so I decided to check them out and see what was available.

So using my Zune Software on my PC, I was able to view the various available apps for the Phone 7. As it turns out, about one fourth of them are simple tip calculators, flashlight, and unit conversion apps. Not really the kind of apps you’d really need or want to pay for! There are a fair number of games, which seems to be the current “best sellers”. Some of the big name apps are available (e.g., Netflix, WeatherChannel, etc) so I would imagine you’ll see a lot of standard, useful apps currently available on the other platforms coming to the Phone 7 soon.

Over the last week I’ve been investigating the Phone 7 SDK tools and it is clear to me that Microsoft has spent a lot of time developing this OS and platform. The apps are based on Silverlight 3 and games on XNA, and both are well established programming environments. The greatest downfall that I see is that Phone 7 doesn’t have a legitimate database system for effectively storing local data, and that may be the Achilles’s Heel for some high-powered apps from being ported over to this platform.


Computer component organization for a neat freak

October 23, 2010

I’m a bit of a neat freak and don’t like clutter around my house and especially in my home office. I can’t control what my kids, dog, and wife do with most of the rooms in our house, but my home office space is my personal kingdom. As such, I don’t like all the wires, cables, power outlets, etc. that come along with all the various computer components and gadgets that I have currently in my office. I was able to reduce much of this by switching to a fully integrated iMac Desktop system, but I still have the following components to deal with:

  1. External backup hard drive (for use with Apple Time Machine Software)
  2. Comcast digital set top box
  3. EyeTV HD video recording device
  4. Touchstone charger with USB cable
  5. Work Laptop charger

Considering that my office has basically a single desk (floating in the center of the room) and a small side-board cabinet just off to my left side, I don’t have a lot of space for electronic accessories. Of course, I could put the  mentioned components inside my side cabinet, however, I didn’t want to purchase extension cables and have them run from the side cabinet over to my iMac system sitting on my main desk. Thus, I designed a method to store all of my components under my main desk.

What I did, was buy a pegboard (normally used for hanging up tools in a garage) and cut it down to a 2-ft by 4-ft piece. Using three small hinges, I attached one end of the pegboard to the bottom front side of my desk so that it could swing up and under my desk. This way, I could easily attach the three previous mentioned computer components to the pegboard using plastic zip ties and swing it up and out of the way under my main desk surface. What is nice, is that I could also neatly wrap and tie all the associated power adapters, cables, and wires to the pegboard so that everything is out of sight. I also included a powerstrip so that all these accessories could be conveniently plugged in under my desk as well.

Now if you walked into my home office you wouldn’t notice any of these extra components connected to my iMac (sans the few USB cables coming from the back of my iMac disappearing around the underside of my desk). If I ever need to check any of these components or add something else, I just swing it down and viola I have instant and easy access.

The under-the-desk trick should work for anyone who has a desk with a reasonably hiding area beneath it. I highly recommend using pegboard as it has pre-drilled holes which the plastic zip ties can conveniently wrap around. And if you’re a super-neat freak you could also strap in a small light which you can turn on to actually see what you have under your desk! 🙂


Switch to an all Mac TV recording system

October 23, 2010

For the last several years I’ve been using Windows Media Center with my Microsoft Vista (and later Windows 7 PC) for recording TV shows using an internal TV Tuner Card.  I had a complete system in place to record my scheduled TV shows and use a nifty utility called DVRMSToolbox to handle the identifying of commercial segments (via the app called ShowAnalyzer), removing of those commercials, and converting the edited show to a WMV file for my Zune HD. Everything was automated, where all I had to do was sync my Zune in the morming and I was good to go!

Since switching to an Apple iMac, the only reason to have my Windows 7 Desktop PC still running is to record my TV shows for my Zune. Since I really wanted to shutdown my Windows machine for good, I decided to switch over to an all-Mac system for recording and processing TV show recordings.

To begin, I purchased a device called EyeTV HD made by Elgato for use with my iMac. This external USB device isn’t a TV Tuner, but instead processes a digital TV signal coming from a digital set-top-box provided by my cable TV provider (Comcast). As such, I needed to upgrade my current digital transport adapter (DTA) box from Comcast to one that can output digital component video. So once having the new digital set top box and the EyeTV HD device, I was able to connect everything up to my iMac for viewing and recording TV shows. Fortunately, the EyeTV HD came with an IR remote changer which I connect to my set top box for changing the TV channels via the EyeTV software.

So in comparison with my old Windows setup, the Mac setup is very similar in capability. Instead of using Windows Media Center, I now use an application called EyeTV3 which came with the EyeTV HD. This new app has similar functionality as the Windows Media Center software, however, it is designed to run like a normal computer application rather than being controlled by a remote control sitting on a couch. It uses programming information obtained from the Internet (via TV Guide) which is free for the first year (subscription fee yearly after the first year). I can search for upcoming shows and flag them for recording. I can also have EyeTV3 automatically convert the recorded TV shows to iPad, iPhone, or iPod format and put the converted files in my iTunes folder for syncing. All if this is handled automatically. Also, if the EyeTV3 app is not running, my Mac system will automatically fire up the app if necessary to record a scheduled TV show.

The one thing I really miss is the commercial removal feature that I had with my Windows 7 setup. Unfortunately, using the EyeTV product doesn’t lend itself for automatically commercial removal. I did find some freeware utilities that center around the Comskip program that identifies commercials in a video file, but there isn’t an automated way of removing the identified commercials from a EyeTV3 recording and convert the video to m4v format for iTunes. I did play around with this method a bit, and was able to accomplish the following:

  1. Setup EyeTV3 to automatically initiate Comskip to scan for commercial while a TV show is recording, and save the markers to a file.
  2. Manually “compact” the recorded TV show to remove the commercials (based on the markers found by Comskip).
  3. Manually initiate an “export” of the compacted video to a m4v file and put the file in the iTunes folder for syncing.

So, I was able to do the commercial editing in a semi-automatic method, but it can’t be done completely automatically as with my Windows 7 setup. Of course, I might be able to automate some of this process better by using  Applescript commands, but that’s something I’m going to really have to dig into and get creative. From what I can tell, there’s no way of easily initiating the compacting of the video file other than from the EyeTV3 interface, so that might be a road block in automating the entire system.

However, I am very pleased with the entire setup and how it all works seamlessly. I can now turn off my Desktop PC permanently, and retire my Zune HD in favor of my new iPod Touch (which I should be getting next week as a birthday present from my wife).


Major Flaw in Windows Phone 7 OS

October 18, 2010

I’ve been investigating the development of apps for the upcoming Windows Phone 7 OS smartphone, and it’s been a dizzy adventure in cyberland. There seems to be lots of bits and pieces of information available on the web, ranging from tutorials, Microsoft training videos, and forum postings. From what I can tell, the Phone 7 OS is a fresh, new start for Microsoft serving as a reboot from their previous (and tired) Windows Mobile 6 OS. I do like the appearance and apparent operation, however, it does have one glaring flaw: No local database storage.

If you look at the Android OS, Palm WebOS, and iOS (iPhone), they all have SQLite available their SQL database storage system. This is a good thing, since SQL is a standard database system that allows for sorting, ordering, etc. of huge amounts of data. The Windows Phone 7 OS, doesn’t have SQLite nor any kind of advanced database system built-in. In several forums, Microsoft MVPs simply shrug this issue off and say, “… just store your data in an XML/CSV file for local storage”, or, “… put your data in the Cloud for storage”. Storing data in the cloud is completely impractical in my opinion, since you would always need an Internet connection to access your data. Thus, local storage is a must and keeping your data in an XML or CSV file is not very efficient for medium to large amounts of data. Also, it’s a real pain to write code to deal with such formatted files.

To me, this is a real blunder right out of the gate that Microsoft needs to fix fast if they want developers to port their existing Android/WebOS/iPhone apps to the Windows Phone 7 platform. Sure, you might be able to use a 3rd-party or open-source C# coding that emulates the SQL database system, but you’ll be reliant on this 3rd-party coding to work and be optimized for robustness. I personally wouldn’t want to be in that position as  a developer, as you’re putting too much trust in a 3rd-party add-on from a company that isn’t all that vested in your app to succeed and work properly. This is something that Microsoft needs to fix and quickly, otherwise you’ll see a very limited number of apps appearing after their initial launch.