BOOTING SDI WITH SYSLINUX

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

On Thu, 30 Dec 2004 09:52:30 -0500, Dietmar wrote:

Hi again Dietmar,

I'm sorry for this inconvenience, but I did this patch for the company in
which I work, and I don't want to cause them problems. In all cases, like
I said, I'm in holidays and don't have those files right now. I really
want to help you, but can't do this beyond the legal stuff, and I was
quite sure that you could find the file by yourself.

Anyway, I'm glad you found the same NTLDR that I use, but as Slobodan
said, I'm afraid that you don't get more success with syslinux than with
the standard solution to boot images > 500 MB, it's a driver issue, and
the driver is not included in the boot loader.

Best regards,

Rémi

 

[KM]

Dietmar,

How about installing the Remote Boot Server and getting the ntldr from the Program Files\Windows Embedded\Remote Boot
Service\Downloads\ntldr directory?
http://www.microsoft.com/downloads/...6f-1f3a-4194-8e26-a4c9653adbc5&displaylang=en


The ntldr length there is exactly 34400h. (don't know what else we could expect )
You will also find startrom.com and startrom.n12 there.

KM

PS. Seems like I missed the entire conversation by taking a nap

 

[KM]

Rémi,

Hi Dietmar,

Do you have problems with syslinux and the SDI patch ?

KM, you seem to confound Linux & syslinux. There is "linux" in syslinux
because it was initially a bootloader to boot linux. But a lot of formats
have been added with time (boot sectors, com & com32 images, disk images,
...) and with the patch, SDI. It is just a quite complete boot loader.
Actually it can be used without anything linux related (a windows tftp
server to boot a xpe client). With the patch, it is just an alternative to
startrom.n12 (/.com).

I see. Thanks for the clarifications!

I am interested in seeing some alternative bootloader solutions as, in my opinion, there is not so many publicly available on the
Windows market.

Since you mentioned you have a "patch" for the startrom.*, did you have a chance to implement a non unicast version (MTFTP, or
etc.)? You'd certaily have to worry about the server side as well.

KM

 

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

Rémi,


I see. Thanks for the clarifications!

I am interested in seeing some alternative bootloader solutions as, in my opinion, there is not so many publicly available on the
Windows market.

Since you mentioned you have a "patch" for the startrom.*, did you have a chance to implement a non unicast version (MTFTP, or
etc.)? You'd certaily have to worry about the server side as well.

KM

Hi KM,

I don't have a patch for startrom.*, I made a patch for syslinux which
makes it an alternative for startrom.* but it is probably what you meant.

I'm not interested at the moment for non unicast transfer, and I have
not so much free time, so it is unlikely that I adapt syslinux to support
it. I can only refer you to H. Peter Anvin (the syslinux developer) who
said:


"As several of you have asked, tftp-hpa isn't likely to support multicast
TFTP any time soon, because such support would require some very
fundamental structural changes which would be very much against the goal
of keeping tftp-hpa "psycho portable", as I like to say.

However, I'm playing with the possibility of supporting multicast TFTP
(the RFC version, not the weird "MTFTP" Intel/PXE version) in PXELINUX. I
originally thought it was impossible without completely breaking the very
nice filesystem abstraction model I had already implemented, but I think I
might be able to deal with that, at least for the large files (kernel and
initrd, as opposed to config and display files.)

However, in order to test this I need a TFTP server which can do TFTP
multicast. Anyone happens to know of one?

For those of you that are interested, the reason this is tricky is that
one of the consequences of using TFTP multicast is that packets arrive out
of order. The current SYSLINUX framework assumes sequential access. To
make matters worse, the kernel file is not only not contiguous in memory,
you don't know what the memory layout will be until you have read the
first block. This can be dealt with by doing large copies, of course, but
that's rather unpleasant. It may very well be the only sane way to deal
with it, however."

and Harry Hoffman added:

"pxelinux works just great with TFTP. Why would you need MTFTP? It would
save some packets if you boot multiple clients at a time, but would need major
effort to get this into pxelinux. Also, if you managed to boot all of your clients
with MTFTP, they would also need to get some file systems (maybe root
filesystems)
via nfs or else, and this definitively doesn't work for nfs, smb and what else you
could think of.

Does anyone know of any other network bootloaders that use multicast? Do many
people see this as a need?

I don't see this as a need, since tftp works even for booting some hundred machines
at a time. You need some sane network structure for this (switched
ethernet with a
higher uplink to the server), but then it's working as it should, and also this should
be standard for modern networks."



So as you can see, it might be supported (however unlikely in the mtftp
form) in the future. It was in 2003, but I don't think there has been
updates for multicast in syslinux since.

Rémi

 

[Dietmar]

Hi Konstantin

thank you for the link.
I dont know whether it is the same ntldr, as I used.
I will prove this whether that one works also with SDI.

Now I gave you the exact description of what ntldr you must use for SDI
with Syslinux. (I have no company)

startrom.com in the same directory as ntldr SEEMS to be ever the same.

ONLY that ntldr works for me:

Windows Embedded SP1 Version 5.1.2600.1106
with ntldr 214016 Byte (34400h)
build 4.September 2002 16:01:46
xpsp1.020828-1920

in directoty C:\...\Windows Embedded\Remote Boot Service\Downloads\ntldr.

I tried so much ntldr's, but that from Saad Syed
I cant found.

Good luck
Dietmar

 

[Dietmar]

Hi all,

The ntdetect.com in the above desribed version of Windows
Embedded in the folder C:\...\Windows Embedded\Remote Boot
Service\Downloads\

DIFFERS from newer Versions of XPe with SP2.

It is 47580 Bytes, (like that in XPProSP1)
and the newer ntdetect.com are 47564 Byte.

Nice to hear from you,
great work Remi

Dietmar

 

[KM]

Rémi,

Thanks for forwarding us the info from the syslinux devs.

I don't have a patch for startrom.*, I made a patch for syslinux which
makes it an alternative for startrom.* but it is probably what you meant.

Yup. That is what I meant.

I'm not interested at the moment for non unicast transfer, and I have
not so much free time, so it is unlikely that I adapt syslinux to support
it. I can only refer you to H. Peter Anvin (the syslinux developer) who said:


"As several of you have asked, tftp-hpa isn't likely to support multicast
TFTP any time soon, because such support would require some very
fundamental structural changes

No doubt.

which would be very much against the goal
of keeping tftp-hpa "psycho portable", as I like to say.

Not sure I understood this. What portablility he is talking about here? (or, better ask, portablility of what?)

However, I'm playing with the possibility of supporting multicast TFTP
(the RFC version, not the weird "MTFTP" Intel/PXE version) in PXELINUX.

Forget about Intel's implementation of the MTFTP in PXE ROM. I does not work as needed on large networks.
I was developed for a small boot program downloads only (and this is clear from Intel protocol draft).

I originally thought it was impossible without completely breaking the very
nice filesystem abstraction model I had already implemented, but I think I
might be able to deal with that, at least for the large files (kernel and
initrd, as opposed to config and display files.)

I don't know anything aobut the syslinux filesystem abstraction model but it definitely can be done for the large files.

However, in order to test this I need a TFTP server which can do TFTP
multicast. Anyone happens to know of one?

There are some MTFTP servers on market. But you will unlikely need those.
The major reason is that you will need to tweak MTFTP protocol (if not rewrite). Current protocol specification is not meant for
multiply synched downloads of large files.
E.g., the paket counter is only 2 bytes (even with MTU 1536 it does not give you much).

So the point is that you will likely have to write your own "MTFTP" server.

For those of you that are interested, the reason this is tricky is that
one of the consequences of using TFTP multicast is that packets arrive out
of order.

Absolutely. But this problem can be fixed by the using an enhanched packet counter.
Or, tagain, in your own protcol version you can establish more proper ways to keep the order.

The current SYSLINUX framework assumes sequential access. To
make matters worse, the kernel file is not only not contiguous in memory,
you don't know what the memory layout will be until you have read the
first block. This can be dealt with by doing large copies, of course, but
that's rather unpleasant. It may very well be the only sane way to deal
with it, however."

and Harry Hoffman added:

"pxelinux works just great with TFTP. Why would you need MTFTP? It would
save some packets if you boot multiple clients at a time, but would need major
effort to get this into pxelinux. Also, if you managed to boot all of your clients
with MTFTP, they would also need to get some file systems (maybe root
filesystems)
via nfs or else, and this definitively doesn't work for nfs, smb and what else you
could think of.

Again, I can't speak for the syslinux, but in Windows world.. You don't need FS support on the client side there from bootloader.

Much more problems appear when you think about diskless stations (not having a temp local storage for the boot).
SDI will not work for you (unless you have amasinly huge RAM on your client device).

I was researching this question a while ago and could not come to a final conclusion.
However, I think if there were a buninesse case (a strong demand) for the multicast support, MS would have already covered it.
(I am still thinking about Windows world, sorry)

I don't see this as a need, since tftp works even for booting some hundred machines
at a time. You need some sane network structure for this (switched
ethernet with a
higher uplink to the server), but then it's working as it should, and also this should
be standard for modern networks."



So as you can see, it might be supported (however unlikely in the mtftp
form) in the future. It was in 2003, but I don't think there has been
updates for multicast in syslinux since.

Yes, I agree. It can be supported but not in the RFC'd MTFTP form.
Either some protocol tweaks need to be done or a new protocol implemented.

I was evaluating some different approaches for multicast downloads of large OS images a while ago (maybe the same 2003) and we (not
only me) came up with prety good specs on the implementation side (I mean we spec'ed protocol, code, bootloader, etc.).
It was quite a long time ago and I do not remember all the details (unless I read through the specs again).

But the point is that it is doable. Since syslinux hs some good improvement on bootloader part, I thought you might have come up
with a multicast solution already.
Anyway, thanks for the dicussion and good info.

Do you have links to syslinux newgrwoups or forums (which searchable engines)? I understand it is under GPL so it should be covered
in public somehow.

Thanks,
Konstantin

 

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

Rémi,

Thanks for forwarding us the info from the syslinux devs.


Yup. That is what I meant.


No doubt.


Not sure I understood this. What portablility he is talking about here? (or, better ask, portablility of what?)


Forget about Intel's implementation of the MTFTP in PXE ROM. I does not work as needed on large networks.
I was developed for a small boot program downloads only (and this is clear from Intel protocol draft).


I don't know anything aobut the syslinux filesystem abstraction model but it definitely can be done for the large files.


There are some MTFTP servers on market. But you will unlikely need those.
The major reason is that you will need to tweak MTFTP protocol (if not rewrite). Current protocol specification is not meant for
multiply synched downloads of large files.
E.g., the paket counter is only 2 bytes (even with MTU 1536 it does not give you much).

So the point is that you will likely have to write your own "MTFTP" server.


Absolutely. But this problem can be fixed by the using an enhanched packet counter.
Or, tagain, in your own protcol version you can establish more proper ways to keep the order.


Again, I can't speak for the syslinux, but in Windows world.. You don't need FS support on the client side there from bootloader.

Much more problems appear when you think about diskless stations (not having a temp local storage for the boot).
SDI will not work for you (unless you have amasinly huge RAM on your client device).


I was researching this question a while ago and could not come to a final conclusion.
However, I think if there were a buninesse case (a strong demand) for the multicast support, MS would have already covered it.
(I am still thinking about Windows world, sorry)


Yes, I agree. It can be supported but not in the RFC'd MTFTP form.
Either some protocol tweaks need to be done or a new protocol implemented.

I was evaluating some different approaches for multicast downloads of large OS images a while ago (maybe the same 2003) and we (not
only me) came up with prety good specs on the implementation side (I mean we spec'ed protocol, code, bootloader, etc.).
It was quite a long time ago and I do not remember all the details (unless I read through the specs again).

But the point is that it is doable. Since syslinux hs some good improvement on bootloader part, I thought you might have come up
with a multicast solution already.
Anyway, thanks for the dicussion and good info.

Do you have links to syslinux newgrwoups or forums (which searchable engines)? I understand it is under GPL so it should be covered
in public somehow.

Thanks,
Konstantin

Hi Konstantin,

I don't think the syslinux mailing list has "searchable" engines. However,
you can download the full archive at this address:

http://www.zytor.com/pipermail/syslinux/

When you have this, you can easily search for anything in it.

regards,

Rémi

 

[KM]

Rémi,

I don't think the syslinux mailing list has "searchable" engines. However,
you can download the full archive at this address:

http://www.zytor.com/pipermail/syslinux/

When you have this, you can easily search for anything in it.

Thanks for the link. Any information is always helpful

Konstantin

 

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

Slobodan, Konstantin, anyone...

Have you an idea why the loader only works with this
version of NTLDR (34400h size) ?
Are you aware of any behaviour difference between the versions ?

I really don't see why it only works with this one; particularly, it
should work with the later versions released with XPe (SP2 and sooner).

Rémi

 

[Slobodan Brcin \(eMVP\)]

Rémi,

You are not using startrom, right?

I have never analyzed how it is implemented, could you provide us with info about your replacement?
I guess that startrom and your patch depend on some mem addresses that are only present in Y2002 ntldr version.

Regards,
Slobodan

 

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

On Fri, 31 Dec 2004 10:37:18 +0100, Slobodan Brcin (eMVP) wrote:

I use the "standard" Saad Syed method.
startrom.com and NTLDR are included in the SDI file. I copy the boot blob
(startrom.com) at 7C00h and call it with the address of the SDI bitwise
ORed with 041h in registers.

Rémi

P.S: perhaps there is a way to call directly NTLDR ?

 

[Slobodan Brcin \(eMVP\)]

Rémi,

I was under impression that you do not use startrom. If you use startrom then you are have two possible choices.
1. That your loader make havoc in processor tables during the protected/user mode switches. That work for some strange case with
original version.
2. That startrom is written in a way that it will jump into certain offset into ntldr.

In second case you can't do anything except to write your own startrom, which I thought that you did.

Now I'm even more interested in reasons why you used syslinux since I do not see reason for it. (Perhaps only better network
support?)

If only reason was that you need to copy data to high memory (>1MB) then there is BIOS call that do exactly that.
If will handle all nasty transitions to protected mode and back, and as long as you are concerned your program will always stay in
user-mode and when you load your image you just do jump to startrom as you described.

You can find source code for loader that fit in MBR area and that can load and execute XPe boot from local storage mediums.

Regards,
Slobodan

 

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

Slobodan,

Rémi,

I was under impression that you do not use startrom. If you use startrom then you are have two possible choices.
1. That your loader make havoc in processor tables during the protected/user mode switches. That work for some strange case with
original version.

I'm not sure about that. I replaced at a moment the protected/user mode
switch by a BIOS call and had the same behaviour. It's strange.

2. That startrom is written in a way that it will jump into certain offset into ntldr.

What exactly startrom is doing ? Do you know a documentation that would
permit me to write my own startrom ? What would be the benefits ?

In second case you can't do anything except to write your own startrom, which I thought that you did.

Now I'm even more interested in reasons why you used syslinux since I do not see reason for it. (Perhaps only better network
support?)

Like I said earlier, loading time is important to me. I get significant
improvements with my loader but my initial aim was to use UDP
fragmentation to transfer the image. However almost all PXE
implementations in network cards are broken and don't handle correctly
fragmented packets. The only (and not really real) working one I found
is the VMware virtual card vmxnet (about 15 seconds to
load a 200 MB image with 65K packets).

Others reasons are custom messages during image loading and the need to be
able to load different SDI files depending of the subnetwork.

More generally, controlling both the client and the server side of the
loading may be useful.

If only reason was that you need to copy data to high memory (>1MB) then there is BIOS call that do exactly that.
If will handle all nasty transitions to protected mode and back, and as long as you are concerned your program will always stay in
user-mode and when you load your image you just do jump to startrom as you described.

You can find source code for loader that fit in MBR area and that can load and execute XPe boot from local storage mediums.

I think I saw this loader. However it seemed to me that it also uses
startrom, am I wrong ?

regards,

Rémi

 

[Slobodan Brcin \(eMVP\)]

Rémi,

I'm using startrom and I do not have any documentation on its implementation.

I just thought that you have disassembled it and made similar replica by yourself.

Regarding the network boot using syslinux make sense.

Regards,
Slobodan

 

[KM]

I basically did similar. Just disassembled startrom.com (n12) and analyzed the code a little bit. It is fairly simple code that
leverages PXE (NetPC ROM) BIOS APIs (and there is existing specification docs on it), TFTP download (through PXE) and check on
ntldr.

I had failed to find any docs on the startrom.*.

Konstantin

 

[=?iso-8859-1?q?R=E9mi_Lef=E8vre?=]

Hi Konstantin,

So because I do in my loader the tftp transfer (with PXE API), perhaps I
could call NTLDR directly ? Do you know if it is called in real or
protected mode and what parameters (processor registers) are passed to it
? I don't have a Windows binaries disassembler at my disposition at the
moment.

Perhaps there are differences in the way to call NTLDR between the
versions that work and the ones that don't...

Cordially,

Rémi

 

[KM]

Rémi,

I don't have that dissembled code in front of me now so I can't send you the exact answers.

When I have a little bit more free time, I will go though the code again and let you know if I can easy extract the information you
need.

IIRC, there was no switch from real to protected mode in the dissembled code.
Also, I don't recall I could find out how they jump to ntldr but I might have not spent enough time reviewing the code that time.
But I saw PXE calls, I think.