The natives are getting restless...

[brucer]

So where are we at with guide data, the current situation is pretty useless and it seems jaidev/timmy are getting nowhere fast.. what exactly is the problem with the scraper? Why not upload the source and document the existing problems so others can try and help - many hands make light work etc.. I have a week off at present and would be happy to take a look..

[number6]

So where are we at with guide data, the current situation is pretty useless and it seems jaidev/timmy are getting nowhere fast.. what exactly is the problem with the scraper? Why not upload the source and document the existing problems so others can try and help - many hands make light work etc.. I have a week off at present and would be happy to take a look..

Have patience my friend.

I know Jaidev [and I guess Tim too] are working as fast as they can. Hey they and us all have regular day jobs as well you know :-)

I've been working with Jaidev on and off as much as possible since before this forum started on sorting this problem out for good.

I know Jaidev [just] returned from holiday.

I have also been out of action for a few weeks over April/May so things slowed down, but we're all back doing our normal jobs now, so...

...I believe we are getting closer to the guide data nirvana we all want.

I guess you realise now, [that along with some of us 'old timers' who realised this quite some time ago] website scraping is not the long term answer it was once hoped to be.

[timmy]

ok current situation for guide data right now isn't great I admit although it's not fair to say we are getting nowhere fast. I am currently building a tvscheduling website which is progressing nicely and has an ETA of around 2-3 weeks. I am not sure where Jaidev is with sat extraction but it will be done at some point.

The tvscheduling site I am building will plug into the sat data eventually but will use existing guide data sources for the foreseable future. I believe the tvscheduling idea will enable us to at least get season information locked down (by user input) and the gaps (ie one-offs) being filled by existing guide data.

I understand your frustration. I've been in your situation before, having other people seemingly less interested in getting guide data but not that willing to offer that knowledge to help others.

I also understand that guide data is the achilles heel for PVR devices and hence i'm working my *** off to get guide data. I just can't be assed to continue to fix broken grabbers when I know better solutions are achievable.

You're welcome to my grabber code (C#/.NET), give me your email address and i'll send you the database and code. The code isn't great tho (needs a few major improvements but does currently work), the problems are both with the sky site (unreliable, badly formed pages, etc) and with my code (not robust enough). The code worked well for a while (like a month or so) but the less time I spent dedicated to the task the less reliable it became to the point where it just wasn't worth it. I'm not the only one, Brandoo has written a grabber also, as have one of the xmltv guys, I think we all have similar problems.

stay tuned for the tvschedules site!! I'm confident it will exceed your expectations in terms of functionality and usefulness.

TiM

[brucer]

Hi Timmy,

I realize that my original message might have sounded a rude, sorry bout that and thanks responding so reasonably ;) I guess I'm just chomping at the bit and wanting to help..

you can reach me at bruce@zitafarm.com and, as C# is my new best friend, I'd be most happy to take look at your code and see if there's anything I can help with..fwiw until you get tvschedules site going.

Bruce

[timmy]

ok done!

I wrote some notes too, I hope it's enough to get you started.
I also have a tvnz grabber too that you're welcome to if your keen.

Good luck :)

[jaidev]

Yes as stated in the announcements thread we are not continuing active development of the scrapper.
We are dedicating our time to better methods of obtaining guide data. Tim is working furiously at the website scheduling idea, I and with the help of number6 are working on the sat grabbing which is coming along nicely.
However developing these new solutions take time we do have full time jobs and other commitments hence why we stopped active development of the grabber, it required precious time which we can now use for our other projects.

Although you are still getting guide data, this was a priority of mine. Being able to still provide guide slices, I am creating slices from the nerdthings grabber. It's not perfect but it'll do till we have our solutions in place. I think modification of the grabber is now a waste of time, we are not going to use it.

[gadgetman]

Ok, I have read and re-read quite a few of these threads, and as a newbie would like to find out where the whole issue is at. From what I understand things are as follows:

* Getting program data is not that hard, and mostly works, either thru scraping or manual input/tweaking

* Getting good slices is proving hard

* The sat data extraction is the way forward in regards to getting core schedule data, but scraping is still used until this is finished.

If I am wrong on the above, please correct me! It is quite hard with so many threads covering basically the same topic. I have also written a scaper uses Sky's website which works well (I did this before I even found this forum).

Also it seems there are quite a few of us with commercial programming backgrounds, with our combined technical knowledge it shoudn't be too hard to get everthing working????

Final part, no-one has ever said what a slice is!!! I guess it is what TIVO's use for scheduling (like if they dialed up as per overseas) but how do they work and get onto the TIVO?

Many thanks!

[timmy]

ok programme info (guide data) and slice is essentially one and the same. The slice is just the format for the Tivo. The tivo is a little fickle with the slices tho, ie negative times and duplicates it doesn't like (which is probably fair enuf).

you say you have a working grabber for sky, congrats. After you've been grabbing for a couple of months you'll realise the sky site is something of a moving target and your grabber will break frequently unless you keep a close eye on it.

[gadgetman]

mmm, I now what you mean about sky being a changing target, I have already had to change my app once, but for what I wanted at the time it worked. I didn't realise (at the time I wrote it) there where so many other people interested in this, of course I didn't realise there was a TIVO community in NZ....... thanks god there is!!!

[stuge]

First off, please don't shoot me - I am an entheusiastic newbie with the glimmer of an idea.

The tivo 'nag' is an annoyance that is managed, I understand, by loading a fake slice with years worth of dummy schedule data. This effectively tricks the tivo into thinking that the whole schedule is under control even if the latest *real* slice only has a few days of data in it.

I see the logic, and have a couple of questions:

Does this mean that there is now a fake channel loaded into the Tivo that shows up when you want to see what is on live TV? If it does, I would think that is a bit annoying.

As an alternative idea, is it possible to artificially fill up the real slices with a few extra weeks of dummy data every time they are posted? That way the Tivo always gets what it needs and you don't have any extra channels that you don't need.

Stuge
:rolleyes:

[number6]

First off, please don't shoot me - I am an entheusiastic newbie with the glimmer of an idea.

The tivo 'nag' is an annoyance that is managed, I understand, by loading a fake slice with years worth of dummy schedule data. This effectively tricks the tivo into thinking that the whole schedule is under control even if the latest *real* slice only has a few days of data in it.

I see the logic, and have a couple of questions:

Does this mean that there is now a fake channel loaded into the Tivo that shows up when you want to see what is on live TV? If it does, I would think that is a bit annoying.

As an alternative idea, is it possible to artificially fill up the real slices with a few extra weeks of dummy data every time they are posted? That way the Tivo always gets what it needs and you don't have any extra channels that you don't need.

Stuge
:rolleyes:

I don't think it needs to be a valid channel (you can see) or anything just a valid program entry for a valid day.

I have a slice file that has 1 program in it for some US TV channel my Tivo has never had in it with a Day of 15000 (Day 15000 = Jan 11 2011). Thats all it needs to shut it up about no guide data.

Why pick day 15000? well its a nice round number and I hope like hell there something better available than 10 year old Tivo techology by 2011!!

Of course your Tivo also gets upset if it hasn't "phoned home" for 30 days or more (even with good guide data in it) so there is a second hack need to fix that part up too.

Most of us have a shell script that resets the "last date called home" entry in the database and thats enough to shut the Tivo up in most cases.

If your Tivo does call the emulator to download guide data, that gets reset each call so you never need the second hack.