guide data/tv schedule website idea

[brucer]

Basically, the guts of this are - let sleeping dogs lie - i.e. don't piss them off to such an extent that they (over) react against you.

Wow.. thanks for the info (I think )

I guess having every Tivo owner thrashing their website with grabbers would fall into the "pissing them off" basket.. hmm, from now on I just use my Tivo to pause live tv

[number6]

while we're on the subject..has there actually been any indication from Sky that they even care? Even when they eventually launch their own pvr service they're not likely to lose much market share to diy tivo enthusiasts..most people, myself included, would prefer to have a reliable pay service than muck around in the linuxy bowels of a hacked tivo

Yes Sky have said that they regard EPG information (i.e. their Program Guide data) as proprietary to Sky and part of the their (Intellectual) Property.

For this reason they refuse to release it to anyone without payment and retain the ownership of the program data in all cases anyway, so you can't do a deal with Sky then sell it to all and sundry without their say so.
They even turned down Microsofts offer to get access to Skys EPG info for their competing product [Xbox TV or whatever it was called].

One guy a few years back did a EPG licensing deal with Sky (before Skys website had listings on it & before Microsoft came-a-calling) to create a "TV scheduling" service which allowed you to bookmark programs (via his website) you wanted to record and the site would email you reminders of stuff to record. He had EPG info from Sky, TVNZ, TV3 and TAB etc but eventually folded the business because he couldn't make money from it.

In his application the website retained the EPG info at all times and the EPG info wasn't onsold to anyone or used in devices like a Tivo.

There were few Tivos in NZ at that stage and in any case, his concept was to try and make money from reselling the Guide data if he could. Basically he was just ahead of the times. Shame really, we could have used him about now... - mght have cost each about $10 per month subscription though :-)

The other point is that yes Sky may well not be too bothered about a bunch of DIYers repackaging their EPG info for "home use" for now.

However, eventually [and sooner than later if the EPG info is good and consistent] this arrangement will become abused as the overseas (and local) eBay/Trademe operators start trying to flog off cheaply sourced US Tivos for hundreds of dollars and saying "all support and Guide data for your Tivo is free from XXXX" - which is what is happening in AUS right now and to be honest there is little they can do about it except lock down the emulator to only dish out guide data to "known tivos" to block the hangers on.

Then suddenly, Skys find that their (precious) guide data is being used to prop up a commercial enterprise [some guy flogging of US Tivos on ebay to NZers], and they will react accordingly - and it will be by going after everyone, not just the abusers.

And you'll also have TV3 and TVNZ breathing heavily in Skys direction since their (precious) EPG data is now also being stolen/used without their permission too (as a result of Skys data being flogged), and Sky get a lot of benefits financial and otherwise from having TVNZ and TV3 on their Pay-TV platform (not least it keeps the government from regulating the entire pay-tv industry here and also keeps TVNZ and TV3 from pushing ahead with Digital Terrestrial TV a competiing platform to Skys Satellite - and Sky have just signed up for 15 more years to Satellite on a (yet to be launched) Optus D1 sat. so they are committed to Sat Pay-TV for now).

So you can see that Sky has a lot to lose and not a lot to gain from letting this sort of thing happen, even in a de-facto kind of way.

In any case, if you disagree, with all this, try ringing up Sky and see if you can get their permission to use their EPG data on your Tivo.
We already tried this approach without much luck some time ago and I doubt it will be any better now.

[timmy]

Ok the legalities of these things are all very interesting and stuff but it still leaves us in a conundrum. I personally think trying to grab from a website is a waste of time. The data isn't very feature full anyway and the problems of trying to maintain a working grabber creates a reliance on programming folk to put in their time to maintain it. I have thought about distributed and proxied grabbing but flagged it... i'm now pretty firm on community built tv listings, which I believe is technically legal.

So my question is, do I continue with my tvschedule site and if so do I use existing online or satelite listings?

If I proceed with the site it is going to raise the profile of our activities, no doubt about it. If I use existing sources (ie web/satelite) of data then it is highly questionable legally so it's a double whammy. If I don't and the guide data is still coming from web/satellite then is the community going to put their backing behind it? If they don't we're left with a tvschedule site that isn't going to be useful and possible increased profile of other guide data grabbing activities.

thoughts people...

[brucer]

how does all this work in Australia? Obviously they've got a much higher profile.. have they been harassed? Is the law different over there or do they have some kind of licensing agreement?

[timmy]

hmm another 2c worth

yeah they are grabbing from websites, their grabbers break, they just keep fixing them, awhile back one site decided they didn't want oztivo grabbing from them so put a stop to it (or something happened anyway), i think they had to source an alternative site, basically they have agreed to not discuss where the guide data comes from and leave it at that. Personally i'm surprised that they manage to get such good guide data from online sources, obviously they're better at coding than me

a while back someone wanted to start a business off the back of oztivo, the community was pretty anti as u can imagine. There is another tivo enthusiast currently trying to create his own PVR service (icetv.com.au?), don't think anyone even knows how he's going about the guide data but he is offering it as a pay service same as tivo so dunno how he is skirting the law...

[number6]

how does all this work in Australia? Obviously they've got a much higher profile.. have they been harassed? Is the law different over there or do they have some kind of licensing agreement?

They are just like us, living in the fear of the coming of the great (free) Guide Data Killing/Publicity monster.

Since there are many more TV channels spread all over Aus and each state seems have their own laws that can change or remove Federal (National) laws, the legal position is pretty much muddier than here especially since by doing stuff across state lines you can circumvent a lot of legal issues.

However, they are finding the recent profile rise due to newspaper articles and rise of the "trying to make a buck of the OzTivo group" operators on Ebay that things may start hotting up there soon.

They do have an advantage in that there are some (Pay to use [ebroadcast]) EPG services in Aus, and in theory if enough Tivo owning folks reached into their wallets each month they could buy a legal EPG feed but the entry fee for that is a little steep (about 10K per month I think). Also, there seems to be a PVR service launching soon in Aus which may also have a positive spin-off benefit to the Tivo guys.

None of this is much use to us though - but they are in the same boat as us more or less, just a little closer to the front of the boat than we are - so that when the onrushing waves hit (us &) them, they may get tipped out first, but we'll no doubt follow suit.

I guess as soon as a proper PVR service hits these shores, the floodgates may finally open up with full EPG data becoming more available, but right now, no-one wants to be the first mover on this so we end up without legal EPG data.

We may have to wait until one of the Aus EPG service operators finally decides to take on NZ as well.

Also, don't forget that its one thing to have EPG data, but it must be up to date and accurate always to be any use.
I guess you've all seen what its like to have semi-ok Guide data - its basically unusable and is more trouble than its worth!

The Aus guys have all found that the existing TV EPG sites they use have lots of problems with bad data - and so a lot of their effort is spent trying to fix dodgy data, let alone trying to obtain reliable clean sources for those hard to get channels. Then you have to align all the data so that the Genres from Source A are consistent with all the other sources they use.
(Curling anyone?)

Really, considering it costs only $20 per month per-Tivo for the US Tivos to have local, accurate guide data, they get a really good deal considering all the hard work that Tivo do keeping the data up to date and accurate.

[gadgetman]

This is aimed more at the others who have developed grabbers. Reading comments here and on other sites about how the grabbers have been written and the fact they break often, I feel I have developed a solution that works differently. I know the whole grabber issue is supposed to be superceeded soon, but in the mean time I thought I should share what I did.

I have a VB6 app with two IE controls, the first loads with the main schedule page from Sky for a given date and time, the DOM of this object is then exaimed and any links to program data are then loaded as a page in the second IE control.

This removes all problems with parsing HTML, and I get all the info about each program, like Genre, date, time, description etc. The only problem is it is slower than parsing pure HTML, one grab of a 3 hour timeslot for all channels takes about 5 minutes, depending on how fast Skys site is. Currently I save this in an Access database (free to any who want a look) but that was just cause I couldn't be bothered at the time creating a SQL database.

This may be a stupid question but would an approach like this give enought raw data for the website idea being promoted on another thread? I have already started expanding my app to pull data for movies from IMDB, more for personal reference than anything.

As for Sky restricting/cutting access to their site, my 2c is that ANY change Sky makes costs money to them. I read an interesting acticle about how there systems all tie together, something like 7 different types, and it sounds VERY complexed (too much so) so I would think they would only stop scrapping as a last resort. Of course legal action is another issue, but they have to prove who got the data! How many people here know about the free anonymous Internet access you can get using GPRS on Vodafone prepay?

Whew! This whole comment has become too long.

[timmy]

hmm, using the DOM is an interesting aproach. nice..
Seriously tho, they can make small changes to those pages very easily, they are after all just coldfusion pages. I know because they have made a number of minor but obvious changes to the html tags and it caused havoc for me until i rewrote my regular expressions to make them more flexible..

The other big issue was reliability with the site, network timeouts, etc..
hmm, encoding to images would be their ultimate f'off to us. Would'nt take much effort on their part and they only need do it randomly on some of the programme info.

hmm, don't fool yourself into thinking it is hard for them to change their site. I don't believe that is the case unless they are even more stupid than I think. I know they have a variety of systems to pull the data thru but the frontend is probably completely seperate and hence able to be modified as much as they want. Presumeably the data is just getting pushed into a database that is accessible to the (external) webserver, thats pretty standard... and from a security standpoint it would'nt be ten foot from the real database(s). In saying that i'm not sure tho because the site was designed by DNZ rather than SKY, they provide a CMS (Content management system) for the site tho whether the listings are a part of that is anybody's guess. who knows who cares...

anyway, i'm thinking to continue development on the tvschedules site and not use any existing guide data sources. That is my feeling right now. I'll gear it up more for general tv watchers (ie non pvr people) as well as support for tivo and xmltv folk. we'll see if it catches any interest... it is good web programming practice for me anyway, not that i dont do enough already. ugh

[brucer]

FWIW, leaving aside all the grabber issues for a second it seems to me we have a couple of realistic choices:

1.we ignore the legal risk and go ahead with a community tvschedule website and emulator and hope nobody notices.

or

2.we all setup our own emulators (or use LOADGUIDE) and roll our own slices and/or share slices via some anonymous P2P service. I'm sure we could safely provide public "for entertainment purposes only" information about setting up an emulator/using LOADGUIDE/using wktivoguide and maybe locating slices on P2P. Beyond that we limit public discussion to technical Tivo issues.

Option 1 is easiest for the non-technical (and therefore open to abuse by commerical interests) but also apparently exposes people like Timmy and Ed to possible legal action.

Option 2 is less convenient but workable, lower profile and much harder to set the lawyers onto.

[gadgetman]

Might be a stupid question, but has anyone asked TVNZ/Sky/etc if there schedule data is available, even for a fee? It must be viable (at some level) for the likes of TV Guide, Listener etc.