Trying to set up emulator in Norway...

[dmicklem]

Hi all,

I'm hoping to get some help setting up an emulator here in Norway (where
there is no official service). I hope this is a good place to post - I figured
the NZ subgroup was the best bet as you seem to have been through setting
stuff up most recently.

So far I have had quite a bit of success. I got a whole bunch of files
from the tivo.canada guys and set up everything as described by them.

The Tivo goes through guided setup fine (via phoning my ISP), and
'sees' the five stations I have set up, with the correct on-screen
names etc. It is able to change channels on the VCR that is acting as
PAL tuner so that is OK too.

So far so good: the Tivo can talk to my server and make sense of what it
hears.

To generate guide data, I've been using mkguideslices.pl which uses
tv_grab (there is a _no version thankfully) to scrape data, filters
it through tv_extractinfo_en, tv_sort and tv_grep. Then through
xmltv2tivo and finally writeguide. At each stage in this process the
data appears to me to be sane.

At the Tivo end, the daily call appears to succeed and the logs suggest
that the guide slice has been downloaded. But now everything falls apart: no
matter how long I wait or what I do the Tivo never recognises that it has
found any guide data.

I'd really appreciate any suggestions for things to look at as I have
been tearing out my hair, redoing everything over again, Clear and
Deleting etc etc without getting any closer to having any clue about
what might be wrong.

I've included heavily trimmed versions of the various intermediate files,
and a summary of some of the info from dumpobj. As far as I can tell all
the numbers match up, but I don't really _know_ what the files should look
like! I hope it isn't too long...

Thanks for any comments/suggestions,

David

I'm using a recent Oztivo Image (1.3 I think). Philips Series 1 with a big
drive added.

The Tivo seems to have interpreted the Headend as follows, and the
numbers seem to match what was set in config files on the server:

SetupID->AVSource->Channels are:
36972->36972/12->36972/13-18.
Looking at just 36972/18:
Number = 1
Station = 37019/-1
Call sign = NRK1
City, Bergen
Country = Norway
Index Path = /StationsTMS/1001:909b /Server/1100001
Name = NRK1
Server Id = 1100001
Server Version =75
State = SH
Tms = 1001
Version = 1
Zipcode = 5010


The headend.txt file looks like this (trimmed to just one station to save space...)

Guide type=3

Station/1/1100001/75 {
TmsId: {1001}
Name: {NRK1}
CallSign: {NRK1}
City: {Bergen}
State: {SH}
ZipCode: {05010}
Country: {Norway}
}
Headend/1/8000000/75 {
TmsHeadendId: {SH05010}
State: {SH}
TimeZone: 9
CityPostalCode: CityPostalCode/2000
PostalCode: {05010}
EncryptionKeys: {199801,1,0x199793}
EncryptionKeys: {199803,1,0x199795}
EncryptionKeys: {199806,1,0x199798}
CommunityName: {Bergen}
CountyName: {Bergen}
Location: {Bergen}
Name: {Providername}
Lineup: Lineup/8000001
Subrecord Lineup/8000001 {
Name: {Linup}
Type: 1
Channel: Channel/10000
}
Subrecord Channel/10000 {
Number: 1
ServiceTier: 1
Station: Station/1/1100001
}
Subrecord CityPostalCode/2000 {
PostalCode: {05010}
CommunityName: {Bergen}
}
}
PostalCode/1/2000000/75 {
Headend: Headend/1/8000000
PostalCode: {05010}
}

The final .xml files (trimmed to one program and one station) looks like:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">

<tv source-info-url="http://fredag.dagbladet.no/tv/" source-data-url="http://fredag.dagbladet.no/tv/" generator-info-name="XMLTV" generator-info-url="http://membled.com/work/apps/xmltv/">
<channel id="1.fredag.dagbladet.no">
<display-name>NRK 1</display-name>
</channel>

<programme start="20040908062800 +0200" stop="20040908093000 +0200" channel="1.fredag.dagbladet.no">
<title lang="no">Frokost-TV</title>
<desc lang="no">Møt dagen med nyheter hver halvtime, aktuelle gjester, filmer for de minste, praktiske tips, underholdningsstoff og mye mer. Programleder:Cille Biermann og Erik Wold.</desc>
</programme>

</tv>

That trimmed .xml file yields this after xml2tv:

Guide type=3

Program/1/150010710/119 {
Title: {Frokost-TV}
Description: {M<F8>t dagen med nyheter hver halvtime, aktuelle gjester, filmer for de minste, praktiske tips, underholdningsstoff og mye mer. Programleder:Cille Biermann og Erik Wold.}
Series: Series/1/100000693
Genre: 1000
}

StationDay/1/200000360/119 {
Day: 12669
# Date: Wed, 08 Sep 2004
Station: Station/1/1100001
Showing: Showing/10
Subrecord Showing/10 {
Station: Station/1/1100001
Program: Program/1/150010710
Date: 12669
Time: 16080
Duration: 10920
# EndTime: 27000
}
}


After running through writeguide and then readguide is looks like this (ie much the same)

Guide type=3

Program/1/150010710/119 {
Title: {Frokost-TV}
Description: {M<F8>t dagen med nyheter hver halvtime, aktuelle gjester, filmer for de minste, praktiske tips, underholdningsstoff og mye mer. Programleder:Cille Biermann og Erik Wold.}
Series: Series/1/100000693
Genre: 1000
}

StationDay/1/200000360/119 {
Day: 12669
Station: Station/1/1100001
Showing: Showing/10
Subrecord Showing/10 {
Station: Station/1/1100001
Program: Program/1/150010710
Date: 12669
Time: 16080
Duration: 10920
}
}

[thomson]

You may want to try and load the slice that you generate manually...

This can be done using:



dbload30.tcl generated.slice
force_index.tcl


This way you can ensure that you are creating a valid slice that will be recognised by your headend. If a manual load does not work then you need to check that your headend and stations match up. You may also like to check your MFS database (via TivoWeb) to see what is being stored in there. Another thing is that you may like to increase the trailing "version number" for some of the records to see if that resolves any issues.

You also seem to be missing a "Series" header in your slice (although you may have just cut it out to save space on the post). It should look something like:


Series/1/100000693/119 {
Genre: 1000
Episodic: 1
Title: {Frokost-TV}
}


I hope some of the above might help... I know nothing about the emulator, but if you post your headend and a sample channel slice then I am more than willing to have a look at them and perform a "sanity" check :)

[dmicklem]

You may want to try and load the slice that you generate manually...

This can be done using:



dbload30.tcl generated.slice
force_index.tcl


This way you can ensure that you are creating a valid slice that will be recognised by your headend. If a manual load does not work then you need to check that your headend and stations match up. You may also like to check your MFS database (via TivoWeb) to see what is being stored in there. Another thing is that you may like to increase the trailing "version number" for some of the records to see if that resolves any issues.

I hope some of the above might help... I know nothing about the emulator, but if you post your headend and a sample channel slice then I am more than willing to have a look at them and perform a "sanity" check :)


Thanks for the suggestion. I'd been meaning to try that but wasn't sure how to get the slice onto the machine. Solved that with a bit of googling :-) Also I had to remember to bring a slice home with me - I can only access the tivo via a serial connection, and it makes accessing the internet at the same time impossible (and makes it hard to access at all - the USB-Serial adaptor doesn't let me switch back to using the computers internal modem without a restart). A side-effect of all this is that I'm not sure I can access TivoWeb, so I have to get all info via bash or tivosh.

Anyway, I downloaded it OK with rz and ran the scripts you mention. It didn't kick out any error messages but I still don't have working guide data (both scripts were completely silent). Curiously, the 'account status' in the sys info screen has switched from 3:Good status to 5: Lifetime Service.

So, I guess there is something screwy in the guide data. As far as I can tell, all the channel/station numbers match up - I posted a somewhat trimmed (single station only) headend.txt and readguided slice (single program on that station) before - are you able to spot anything in that or should I post longer samples? What is the generally accepted way to post sample text/binary files here?

Thanks again,

David

[thomson]

Anyway, I downloaded it OK with rz and ran the scripts you mention
If you have access to a web site with the file on it then you can also download the file from the Tivo using:


wget http://my.external.com/generated.slice


are you able to spot anything in that or should I post longer samples?
Other than the (before) mentioned missing "Series" entry they seemed okay.


What is the generally accepted way to post sample text/binary files here?

Just attach them to a message... the site will recognise only a few formats so you tend to have to "zip" them up. Try and make them small, but if you can upload it, then I will have a look at it. Also... it's a bit of a long shot, but you may also like to try the following:


touch /var/timestamp

[timmy]

To generate guide data, I've been using mkguideslices.pl which uses
tv_grab (there is a _no version thankfully) to scrape data, filters
it through tv_extractinfo_en, tv_sort and tv_grep. Then through
xmltv2tivo and finally writeguide. At each stage in this process the
data appears to me to be sane.


hmm I know Jaidev and I pulled our hair out trying to get xmltv2tivo to work and had no luck tho obviously that is what dem canadian folks are using so obviously it works..

try creating a slice with just one channel and just one day of data and check the data to make sure it is good..

dunno dude..

TiM

[dmicklem]

Other than the (before) mentioned missing "Series" entry they seemed okay.



Doh! I swear I read your message 5 times and managed to skip over that every time!

Interestingly enough there _are_ no Series entries. Maybe a bug in xmltv2tivo? Would that be likely to break it?

David

[dmicklem]

hmm I know Jaidev and I pulled our hair out trying to get xmltv2tivo to work and had no luck tho obviously that is what dem canadian folks are using so obviously it works..

try creating a slice with just one channel and just one day of data and check the data to make sure it is good..

dunno dude..

TiM

Well that in itself is interesting. The canada CVS has two alternatives: p****xmlguide and xmltv2tivo. I only picked the latter because it was easier to get running (p****xmlguide kept running into unfulfilled dependencies). Guess I'll spend time trying to get p****xmlguide running instead. Do you guys use p****xmlguide or something else entirely?

As noted above to thomson, xmltv2tivo doesn't appear to generate Series records which could (??) be a reason it is broken? I'd better make sure I have the latest version.

David

[dmicklem]

Do you guys use p****xmlguide or something else entirely?

David


Heh?? What kind of craziness is this?? It won't let me refer to the place the sun don't shine even in the middle of another word?? (At least it is the UK form it objects to, or is *** a problem too?)

[timmy]

quite a few words like that, haha

We use Warren Toomey's WriteGuide version. Warren is the main oztivo honcho...

http://minnie.tuhs.org/Programs/Wktivoguide/README.html

I'm not sure if that page has a download link, it should be in the oztivo files area if not.

The problem for you is you are getting the guide data from xmltv. You will need to write a routine to convert it into Warren's writeguide format..
The format of these files is documented in the recent guide-data thread in the guide-data forum. It's pretty similar (simpler) to the original writeguide..

hope it helps,

ps how good is the Norway XMLTV data? Know anything about their PVR scene?

TiM

[dmicklem]

quite a few words like that, haha

We use Warren Toomey's WriteGuide version. Warren is the main oztivo honcho...

http://minnie.tuhs.org/Programs/Wktivoguide/README.html

I'm not sure if that page has a download link, it should be in the oztivo files area if not.

The problem for you is you are getting the guide data from xmltv. You will need to write a routine to convert it into Warren's writeguide format..
The format of these files is documented in the recent guide-data thread in the guide-data forum. It's pretty similar (simpler) to the original writeguide..

hope it helps,

ps how good is the Norway XMLTV data? Know anything about their PVR scene?

TiM

Well, I don't have much of a benchmark to measure the XMLTV data against, and I haven't yet tried to maximise it: at least for the time being I'd be satisfied with just the program titles and times. The site that tv_grab scrapes does seem to have quite extensive info, so perhaps with a bit of extra hacking I might be able to get quite a bit. I don't actually know what different fields the tivo is equipped to understand - I suppose that is outlined in the thread you mention?

I don't know anything at all about the norwegian PVR scene. There must be something because someone wrote tv_grab_no. My Norwegian is improving daily, but its still not good enough to do a convincing google for norwegian PVR sites in Norwegian, let alone understand all the esoteric jargon they are likely to be using.

I'm away from home this weekend, but I'm itching to find out whether repairing the lack of Series data will have made my slices work. It turns out that there is a patch for xmltv2tivo that fixes this but never made it into the canadian CVS - once I knew that the data was missing I was able to find an informative post and link in their archives. If it doesn't work, I'll try and get p****xml to work (which means persuading a very recalcitrant perl module to install).

Failing that, I guess I need to trawl the thread you mention for details of the format and work out how to translate the data I have appropriatly. Was the thread in the NZ-specific guide data forum?

David

[dmicklem]

Hurrah! Problem solved.

I applied the series.patch patch (using -R) (available from forums.tivocanada.com) to xmltv2tivo, and recompiled. The generated text slices now contain Series data records, and the resulting writeguide slices are correctly downloaded and interpreted by the Tivo.

Thanks timmy and thomson for the good advice, especially thomson for spotting that the Series records were missing.

David

[thomson]

Thanks timmy and thomson for the good advice, especially thomson for spotting that the Series records were missing.

No worries David. I have been playing with the headend and listing slices for a while. I use to create my own, but now have a process whereby I take a complete listing slice and "massage it" so that it only contains the channels that I am interested in. Hence I have a pretty good idea of what needs to be in a slice ;)

SideNote: The Series entries have been required for the TiVo software since version 2.5, so the chances are that the tools you were using were written quite a while ago.