guide data/tv schedule website idea

[timmy]

hmm, another problem with the sky site is that some channels are updated less frequently than others.
For example tv3,c4,newsaus,prime, etc. These channels only seem to have a weeks worth of data yet the
other channels have upto 3 weeks. FYI the site seems to get updated around the first or second week of the month.

So basically you have to regrab more frequently yet the structure of the pages makes this very download intensive as each programme info is on a seperate page and you can't easily just scrape for a select channel. Even if you can pull out the programme ID's for that channel, ie matching up the table tags and pulling out all of the programme id's for that channel you still have the problem of hitting those pages, eg in two hr intervals thats 12 page requests per day. 7x12=84 pages just to figure out what programme id's you want... annoying. easily doable of course...

I know those programme ID's are in sequence but buggered if i wanted to figure out how the sequences were being allocated.

oh and yes we (well not me personally) have asked sky and they have been less than helpful. and those that do get pay for their listings cannot share them of course... that would be too easy my friend

[gadgetman]

If only sky would setup an FTP server they wouldn't have all our scrappers hitting them, there site would be faster, we would get all the data we need, and we would all live happily ever after!!!

[timmy]

well it was very nearly like that actually. they had an xml file available for download for a while, unfortunately it wasn't meant to be public so they password protected it...

www.sky.co.nz/data

welcome to run a password list over it if u want who knows might get lucky

[deanm]

does anyone subscribe to Sky watch.. would that be considered as paying for Sky EPG? It would seem a reasonable argument..

Now say I take that magazine I paid for.. and employ a data entry clerk to key the data into a database for my personal use.. nothing fancy.. just time.. date.. program title.. (which would be pretty damn cheap and quick IMHO and in terms of copyright would consitute 'Fair Use' of data i've paid for) and my personal database adds to the 'base' data from an online source like IMDB or TVTome.. or even a *cough* NZ based data source am i breaking the law yet?

OK.. i might be wrong here (its been know to happen ), but say individual TiVO users.. all 'armed' with Skywatch subscriptions form a 'hobbyist embedded linux device club'.. and pool resources.. yeah.. yeah.. its a tad simplistic, but you get where i'm going with it. right?

This doesn't address free-to-air data.. or TiVO users without SKY.
It would also mean access to the emulator and 'club' data would only be via paid membership.. even though the club is a non-profit organisation.. there are the basic running costs and the data entry clerk to consider.

While old school.. and VERY low tech.. its also low profile.
No IP to track from grabbing.. just a bunch of geek hobbyists doing there thing.. and feeding a data entry clerk. ehehe

[thomson]

and in terms of copyright would consitute 'Fair Use' of data i've paid for)

The problem here is that we do not have the American 'Fair Use' system... we use the United Kingdom 'Fair Dealing' which is a little different and does not really include reasonable use... Essentially, in this country, if it is copyright then you can not copy it. The (reasonable use) loopholes that appear in the 'Fair Use' system are not openly available to us.

[deanm]

bugger!

OK.. while it is a flawed plan.. it would seem to be a lot less 'trauma' than our current situation.

[gadgetman]

Whilst I now think the idea of a website to enter data is a good way forward (it gives information such as seasons that are unavilable elsewhere), if people want to persure the scrapping idea, there is an easy way to do it totally anonymously using GPRS on Vodafone. Plus scrapping COULD be used as a basis for the website programming.

Because Vodafone become an ISP when using GPRS and prepay involves no disclosure of personal information, you can connect to the 'net without ANY identifying information being available, and then to can scrape away, the IP address is dynmanic as well so unless Sky block the whole address range Vodafone use, it is an idea going forward.

[number6]

Whilst I now think the idea of a website to enter data is a good way forward (it gives information such as seasons that are unavilable elsewhere), if people want to persure the scrapping idea, there is an easy way to do it totally anonymously using GPRS on Vodafone. Plus scrapping COULD be used as a basis for the website programming.

Because Vodafone become an ISP when using GPRS and prepay involves no disclosure of personal information, you can connect to the 'net without ANY identifying information being available, and then to can scrape away, the IP address is dynmanic as well so unless Sky block the whole address range Vodafone use, it is an idea going forward.

I don't think any one here should be that worried about being tracked by Sky/TVNZ etc when downloading the EPG info from their website, after all its why its up there in the first place - to be accessed.

The real problem is that Sky and others can change their website anytime they like for all sorts of reasons [not just to do with blocking downloaders] and whenever this happens it goodnight nurse to the current web-site scraper.

So then someone has to spend hours working out why its broken then fixing it, until that website or another one changes.

Yes, they can also block accessors by IP address, but in the days of ADSL and Dynamic IP addresses its no more of a issue for Sky to block all Xtra IP addresses than it is to block all of Vodaphones if they choose.

No-one said that owning a Tivo without a paid for reliable service was going to be easy - this is "ragged edge" stuff here.

As far as using Vodaphones GPRS service to download websites, the cost of that would soon mount up I'm sure beyond paying someone to sit down and key the TV listings from your Skywatch into your Tivo - and all without the hassle of needing to have web-site scrapers in the first place.
You'd have to repeat it for TV1,2, 3 and Prime as well, and if you live in Wellington, do the same for Saturn TV.

And at the end of the day its to no more legit to do that than the web-site scraping approach.

The long term solution is to have some decent competition out there in the Pay TV market.

The best longer term solution is to get Sky or whatever launching a Tivo-like service of their own as this will need (reliable & constantly fed) EPG data to work and then you can make use of that stream somehow to feed your Tivos guide data habbit.

Right now the whole guide data thing is quite limited by the TV Networks refusal to make Guide data available for more than about 7 days ahead.

The Tivo works best with at least 10 days or more of Guide data to work with.

Skys EPG (on their decoder) seldom goes more than 7 days ahead anyway (it used to go a whole 10-14 days when it started).
TVNZ is not much better either.

[brucer]

Vodafone GPRS is slow and obscenely, revoltingly, disgustingly expensive at $10 per MB for casual use (I use it sparingly sometimes with my Treo PDA), there are hundreds of free anonymous proxy servers out there if you're worried about IP logging (although scraping is dramatically slower via most proxies)..

The Tivo works best with at least 10 days or more of Guide data to work with.

Why is that?

The best longer term solution is to get Sky or whatever launching a Tivo-like service of their own

Of course it will still be illegal but the difference will be they would actually have a good reason to crack down to protect their own PVR service.

It's all very depressing

[brucer]

btw, with all the "criminal intent" expressed in this thread it might pay to delete it once we're done