Difficulties in interoperating with Windows

Difficulties in interoperating with Windows

[Andy Jeffries]

IANAL, but I wondered why there is such difficulty with making Linux work
completely with Windows systems?  I guess part of it may be that Windows
is closed source but as reverse engineering for interoperability is legal
in the UK (regardless of what the End User License states), is the problem
that it is difficult to read the Assembly easily?

Is there not a project on Linux to convert assembly back to C?   Would
this be exceptionally hard?

Just wondered why Linux has struggled for a while to interoperate with
Windows completely...

Cheers,


-- 
Andy Jeffries                   | Scramdisk Linux Project
http://www.scramdisklinux.org   | Lead developer

"testing? What's that? If it compiles, it is good, if it boots up 
 it is perfect."
  --- Linus Torvalds

Re: Difficulties in interoperating with Windows

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> 
> IANAL, but I wondered why there is such difficulty with making Linux work
> completely with Windows systems?  I guess part of it may be that Windows
> is closed source but as reverse engineering for interoperability is legal
> in the UK (regardless of what the End User License states), is the problem
> that it is difficult to read the Assembly easily?

That is not reverse engineering - (at least not MY understanding) - you are
re-translating a copyrighted work. If the translation back into the binary
form creates the same binary then you have an exact translation. You also
have a lawsuit pending. Otherwise you could just disassemble the entire
windows OS, claim it as your "re-engineered source", and sell/publish it.

This is not legal in most locations.

Reverse engineering is taking the published specifications, creating software
that should function in an equivalent manner. Testing it is difficult since
you have to be careful not to import patented/copyrighted algorithms. You have
to compair the inputs/outputs for the software with the inputs/outputs of the
original.

The end result is usable. Linux itself can be considered a reversed engineered
UNIX.

At least a sufficient amount of the specifications are available.

The problem with most M$ software is that the published specifications are not
complete, access to the inputs are not always available (it is ALSO covered
by the proprietary/trade secrets/other restrictions). Sometimes the output
is not available (at least in some countries - DMCA again).

This is also the problem with some device interfaces. If the company claims
"trade secret" or "propriatary information" then the published specifications
will not be usable when reverse engineering device drivers. This problem has
been attacked in the past as "it SHOULD work like this, if I can find a 
register that looks like what I expect, then it MIGHT work"... Some IDE
controllers were originally developed this way - at least one video board
was done this way. (I think even a PCI bridge was deciphered this way)

> Is there not a project on Linux to convert assembly back to C?   Would
> this be exceptionally hard?

Not hard - just illegal when using it to disassemble proprietary software.
Debuggers do this very frequently, to the point that I would say "all the
time" except for debuggers of interpreted languages.

> Just wondered why Linux has struggled for a while to interoperate with
> Windows completely...
> 
> Cheers,
> 
> 
> -- 

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Difficulties in interoperating with Windows

[Andy Jeffries]

> > I guess part of it may be that Windows
> > is closed source but as reverse engineering for interoperability is
> > legal in the UK (regardless of what the End User License states), is
> > the problem that it is difficult to read the Assembly easily?
>
> That is not reverse engineering - (at least not MY understanding) - you
> are re-translating a copyrighted work. If the translation back into the
> binary form creates the same binary then you have an exact translation.

But would it?  If you disassemble part/all of Windows and use the code to
understand how it works, then use this to create a specification and write
code to use that specification, there should be no problem?

> You also
> have a lawsuit pending. Otherwise you could just disassemble the entire
> windows OS, claim it as your "re-engineered source", and sell/publish
> it.
>
> This is not legal in most locations.

Correct, but I'm not talking about recompiling Windows and selling it, I'm
talking about decompiling it and using the decompiled source to make Linux
work better with it.  That is completely legal.

> Reverse engineering is taking the published specifications, creating
> software that should function in an equivalent manner.

I disagree with that definition and agree with this one:

First result for a www.google.com search on "definition reverse engineer"

http://whatis.techtarget.com/definition/0,289893,sid9_gci507015,00.html

:Software reverse engineering involves reversing a program's machine code
:(the string of 0s and 1s that are sent to the logic processor) back into
:the source code that it was written in, using program language
statements.:Software reverse engineering is done to retrieve the source
code of a:program because the source code was lost, to study how the
program:performs certain operations, to improve the performance of a
program, to:fix a bug (correct an error in the program when the source
code is not:available), to identify malicious content in a program such as
a virus, or:to adapt a program written for use with one microprocessor
for use with a:differently-designed microprocessor. Reverse engineering
for the sole:purpose of copying or duplicating programs constitutes a
copyright:violation and is illegal. In some cases, the licensed use of
software:specifically prohibits reverse engineering.

> The problem with most M$ software is that the published specifications
> are not complete, access to the inputs are not always available (it is
> ALSO covered> by the proprietary/trade secrets/other restrictions).
> Sometimes the output is not available (at least in some countries - DMCA
> again).

While I agree about proprietary/trade secrets may be a grey area, where
you have the express legal right to reverse engineer a software product
for the purposes of interoperability surely that is final.  As the
contract would have been between Microsoft UK and you (note I'm only
discussing the UK and we don't have an equivalent of the DMCA here)

> > Is there not a project on Linux to convert assembly back to C?   Would
> > this be exceptionally hard?
>
> Not hard - just illegal when using it to disassemble proprietary
> software. Debuggers do this very frequently, to the point that I would
> say "all the time" except for debuggers of interpreted languages.

Depends where you are in the world I guess.  I am specifically (in the UK)
given the right to do this.

Just interested in opinions on this...


--
Andy Jeffries
Head of Web Development
Internet Assist Ltd

Tel : +44 (0)208 547 3700     Fax   : +44 (0)208 547 3600
Web : http://www.i-a.co.uk    Email : andy@i-a.co.uk

"Helping business achieve quality and cost effective Internet Services."

------------------------------------------------------------------------
The information in this message is confidential and is intended for the
addressee only. If you have received this message in error it must be
deleted and the sender notified. The views expressed in this message
are personal and not necessarily those of Internet Assist Ltd unless
explicitly stated.
------------------------------------------------------------------------

Re: Difficulties in interoperating with Windows

[Jesse Pollard]

> > > I guess part of it may be that Windows
> > > is closed source but as reverse engineering for interoperability is
> > > legal in the UK (regardless of what the End User License states), is
> > > the problem that it is difficult to read the Assembly easily?
> >
> > That is not reverse engineering - (at least not MY understanding) - you
> > are re-translating a copyrighted work. If the translation back into the
> > binary form creates the same binary then you have an exact translation.
> 
> But would it?  If you disassemble part/all of Windows and use the code to
> understand how it works, then use this to create a specification and write
> code to use that specification, there should be no problem?

As long as someone ELSE does the developement (this is the "clean room"
developement that lawyers like for the defence - it must also be fully
documented). Second, you STILL have the problem of "trade secrets" being
appropriated, even if unwittingly.

> > You also
> > have a lawsuit pending. Otherwise you could just disassemble the entire
> > windows OS, claim it as your "re-engineered source", and sell/publish
> > it.
> >
> > This is not legal in most locations.
> 
> Correct, but I'm not talking about recompiling Windows and selling it, I'm
> talking about decompiling it and using the decompiled source to make Linux
> work better with it.  That is completely legal.

Not really - M$ will come after you. That's why the problems with NTFS still
exist - the people that were working on it (even in a "clean room") had to
desist. They (as I understand it) eventually dropped their M$ software. And
NTFS is still read-only.

> > Reverse engineering is taking the published specifications, creating
> > software that should function in an equivalent manner.
> 
> I disagree with that definition and agree with this one:
> 
> First result for a www.google.com search on "definition reverse engineer"
> 
> http://whatis.techtarget.com/definition/0,289893,sid9_gci507015,00.html
> 
> :Software reverse engineering involves reversing a program's machine code
> :(the string of 0s and 1s that are sent to the logic processor) back into
> :the source code that it was written in, using program language
> statements.:Software reverse engineering is done to retrieve the source
> code of a:program because the source code was lost, to study how the
> program:performs certain operations, to improve the performance of a
> program, to:fix a bug (correct an error in the program when the source
> code is not:available), to identify malicious content in a program such as
> a virus, or:to adapt a program written for use with one microprocessor
> for use with a:differently-designed microprocessor. Reverse engineering
> for the sole:purpose of copying or duplicating programs constitutes a
> copyright:violation and is illegal. In some cases, the licensed use of
> software:specifically prohibits reverse engineering.

And M$ will go after you because of the last two sentences. Especially the
"duplicating programs" part. They will (have?) claimed that duplicating
NTFS functionality is not legal. (I think Jeff Merkey was the one doing
this - He should the one to really comment on the problems he had with M$).

Also note - none of that definition addresses the ability to publish the
results.

> > The problem with most M$ software is that the published specifications
> > are not complete, access to the inputs are not always available (it is
> > ALSO covered> by the proprietary/trade secrets/other restrictions).
> > Sometimes the output is not available (at least in some countries - DMCA
> > again).
> 
> While I agree about proprietary/trade secrets may be a grey area, where
> you have the express legal right to reverse engineer a software product
> for the purposes of interoperability surely that is final.  As the
> contract would have been between Microsoft UK and you (note I'm only
> discussing the UK and we don't have an equivalent of the DMCA here)

Yes - but any software that is to be included in the Linux kernel must also
be legal everywhere (well, especially the USA). Just look at the problems
DECSS has had. and the cross agreements between the US and Europe are
beginning to force some agreement to the limits on revers engineering. Check
the refusal of A.Cox to publish explainations of some security patches due
to the DMCA.

M$ has a VERY long arm, and LOTS of high priced lawyers.

> > > Is there not a project on Linux to convert assembly back to C?   Would
> > > this be exceptionally hard?
> >
> > Not hard - just illegal when using it to disassemble proprietary
> > software. Debuggers do this very frequently, to the point that I would
> > say "all the time" except for debuggers of interpreted languages.
> 
> Depends where you are in the world I guess.  I am specifically (in the UK)
> given the right to do this.

But you are not allowed to republish.

It also depends on the software - the way M$ is going, they may build in
a detection for any access to the OS binaries (checkout that DRM patent).
If it isn't from an approved program, the system just may wipe itself out
(assuming it is running :-)

> Just interested in opinions on this...

Sure thing.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Difficulties in interoperating with Windows

[Andy Jeffries]

On Wed, 9 Jan 2002 10:04:11 -0600 (CST), "Jesse Pollard"
<pollard@tomcat.admin.navo.hpc.mil> wrote:
> > But would it?  If you disassemble part/all of Windows and use the code
> > to understand how it works, then use this to create a specification
> > and write code to use that specification, there should be no problem?
> 
> As long as someone ELSE does the developement (this is the "clean room"
> developement that lawyers like for the defence - it must also be fully
> documented).

Hmmm, I don't know about that, as long as the (source) code is different,
I don't think it can be argued that it was copied not created.  But that's
probably a legal battle that no-one would want to get in to.

> > Correct, but I'm not talking about recompiling Windows and selling it,
> > I'm talking about decompiling it and using the decompiled source to
> > make Linux work better with it.  That is completely legal.
> 
> Not really - M$ will come after you. That's why the problems with NTFS
> still exist - the people that were working on it (even in a "clean
> room") had to desist. They (as I understand it) eventually dropped their
> M$ software. And NTFS is still read-only.

Are they US based developers?

> > Reverse engineering for the sole:purpose of copying or duplicating
> > programs constitutes a copyright:violation and is illegal. In some
> > cases, the licensed use of software:specifically prohibits reverse
> > engineering.
>
> And M$ will go after you because of the last two sentences. Especially
> the "duplicating programs" part. They will (have?) claimed that
> duplicating NTFS functionality is not legal. 

But the first of your two chosen sentences seems to read as
copy/duplicating in the sense of piracy.  Obviously as it isn't 100%
clear, then it would be a possible legal case for Microsoft, but to be
honest I can't see the courts going with it.  Otherwise there would only
be one product of each particular type of software.

As to the second: under UK law any license which tries to restrict the
lawful users ability to decompile the product is expressly void.  They
cannot enforce that portion of the contract under UK law (which a UK
citizen buying Windows in the UK would come under).

> (I think Jeff Merkey was
> the one doing this - He should the one to really comment on the problems
> he had with M$).

I certainly would be interested in hearing his comments...is he here and
watching this thread? :-)

> Also note - none of that definition addresses the ability to publish the
> results.

OK, I understand not publishing the decompiled code, but what would be the
problem is publishing your code.

> Yes - but any software that is to be included in the Linux kernel must
> also be legal everywhere (well, especially the USA). Just look at the
> problems DECSS has had. and the cross agreements between the US and
> Europe are beginning to force some agreement to the limits on revers
> engineering. 

Maybe true, but surely though if the decompiling was done outside of the
US and the decompiled code not taken to the US, then the new code is
purely code that works with M$ products.

> Check the refusal of A.Cox to publish explainations of some
> security patches due to the DMCA.

I wonder AC's refusal was more of a statement on the ludicrousness of the
law rather than actually required.

> > Depends where you are in the world I guess.  I am specifically (in the
> > UK) given the right to do this.
> 
> But you are not allowed to republish.

Where does it say this?

> It also depends on the software - the way M$ is going, they may build in
> a detection for any access to the OS binaries (checkout that DRM
> patent). If it isn't from an approved program, the system just may wipe
> itself out (assuming it is running :-)

Hmm, an interesting point...although they'd be hard pressed to do that
when it's decompiled under Linux (extracted from the CAB files off the CD
and decompiled using a Linux tool).

> > Just interested in opinions on this...
> 
> Sure thing.

And thanks so far...

Must remember not to use my work (i-a) address in posts though...

Cheers,



-- 
Andy Jeffries                   | Scramdisk Linux Project
http://www.scramdisklinux.org   | Lead developer

"testing? What's that? If it compiles, it is good, if it boots up 
 it is perfect."
  --- Linus Torvalds

Re: Difficulties in interoperating with Windows

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> On Wed, 9 Jan 2002 10:04:11 -0600 (CST), "Jesse Pollard"
> <pollard@tomcat.admin.navo.hpc.mil> wrote:
> > > But would it?  If you disassemble part/all of Windows and use the code
> > > to understand how it works, then use this to create a specification
> > > and write code to use that specification, there should be no problem?
> > 
> > As long as someone ELSE does the developement (this is the "clean room"
> > developement that lawyers like for the defence - it must also be fully
> > documented).
> 
> Hmmm, I don't know about that, as long as the (source) code is different,
> I don't think it can be argued that it was copied not created.  But that's
> probably a legal battle that no-one would want to get in to.

Yup - there are too many source code manglers that can make what appears to 
be significant changes that do nothing more that change field names, structure
names, and limited re-ordering of statements.

> > > Correct, but I'm not talking about recompiling Windows and selling it,
> > > I'm talking about decompiling it and using the decompiled source to
> > > make Linux work better with it.  That is completely legal.
> > 
> > Not really - M$ will come after you. That's why the problems with NTFS
> > still exist - the people that were working on it (even in a "clean
> > room") had to desist. They (as I understand it) eventually dropped their
> > M$ software. And NTFS is still read-only.
> 
> Are they US based developers?

I think they were/are.

> > > Reverse engineering for the sole:purpose of copying or duplicating
> > > programs constitutes a copyright:violation and is illegal. In some
> > > cases, the licensed use of software:specifically prohibits reverse
> > > engineering.
> >
> > And M$ will go after you because of the last two sentences. Especially
> > the "duplicating programs" part. They will (have?) claimed that
> > duplicating NTFS functionality is not legal. 
> 
> But the first of your two chosen sentences seems to read as
> copy/duplicating in the sense of piracy.  Obviously as it isn't 100%
> clear, then it would be a possible legal case for Microsoft, but to be
> honest I can't see the courts going with it.  Otherwise there would only
> be one product of each particular type of software.
> 
> As to the second: under UK law any license which tries to restrict the
> lawful users ability to decompile the product is expressly void.  They
> cannot enforce that portion of the contract under UK law (which a UK
> citizen buying Windows in the UK would come under).
> 
> > (I think Jeff Merkey was
> > the one doing this - He should the one to really comment on the problems
> > he had with M$).
> 
> I certainly would be interested in hearing his comments...is he here and
> watching this thread? :-)
> 
> > Also note - none of that definition addresses the ability to publish the
> > results.
> 
> OK, I understand not publishing the decompiled code, but what would be the
> problem is publishing your code.

Trade secrets, patented algorithms, DMCA ... I'm sure the lawyers can find
something.


-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Difficulties in interoperating with Windows

[Alan Cox]

> for the purposes of interoperability surely that is final.  As the
> contract would have been between Microsoft UK and you (note I'm only
> discussing the UK and we don't have an equivalent of the DMCA here)

I would not perform such work in the united kingdom. Perform it in a free
country. Reverse engineering that might offend a large corporation in the UK 
is only viable if you have five million pounds to hand, a law firm and a
year to kill.

Re: Difficulties in interoperating with Windows

[Andy Jeffries]

On Wed, 9 Jan 2002 16:22:48 +0000 (GMT), "Alan Cox"
<alan@lxorguk.ukuu.org.uk> wrote:> > for the purposes of interoperability
surely that is final.  As the> > contract would have been between
Microsoft UK and you (note I'm only> > discussing the UK and we don't have
an equivalent of the DMCA here)> 
> I would not perform such work in the united kingdom. Perform it in a
free> country. Reverse engineering that might offend a large corporation
in the UK> is only viable if you have five million pounds to hand, a law
firm and a> year to kill.

I still don't see why you'd need a lawyer.  They could say all they want
and hire all the lawyers they'd want...at the end of the day, could you
just go to court and say:

According to the Copyright, Designs and Patents Act 1988 as amended by the
Copyright (Computer Programs) Regulations 1992 under section 50B, as
detailed below:

Decompilation.
50B.(1)  It is not an infringement of copyright for a lawful user of a
copy of a computer program expressed in a low level language?
      (a) to convert it into a version expressed in a higher level
language, or
      (b) incidentally in the course of so converting the program, to copy
it,
(that is, to "decompile" it), provided that the conditions in subsection
(2) are met.

    (2)  The conditions are that?
      (a) it is necessary to decompile the program to obtain the
information necessary to create an independent program which can be
operated with the program decompiled or with another program ("the
permitted objective"); and
      (b) the information so obtained is not used for any purpose other
than the permitted objective.

    (3)  In particular, the conditions in subsection (2) are not met if
the lawful user?
      (a) has readily available to him the information necessary to
achieve the permitted objective;
      (b) does not confine the decompiling to such acts as are necessary
to achieve the permitted objective;
      (c) supplies the information obtained by the decompiling to any
person to whom it is not necessary to supply it in order to achieve the
permitted objective; or
      (d) uses the information to create a program which is substantially
similar in its expression to the program decompiled or to do any act
restricted by copyright.

    (4)  Where an act is permitted under this section, it is irrelevant
whether or not there exists any term or condition in an agreement which
purports to prohibit or restrict the act (such terms being, by virtue of
section 296A, void).


The law seems pretty damn clear on this issue (in fact Subsection 3.c
sounds like I could quite happily send it to someone in the USA if they
can do the work better than I).

Again, just looking for thoughts...although I hope no-one else uses my i-a
address ;-)

Cheers,

-- 
Andy Jeffries                   | Scramdisk Linux Project
http://www.scramdisklinux.org   | Lead developer

"testing? What's that? If it compiles, it is good, if it boots up 
 it is perfect."
  --- Linus Torvalds

Re: Difficulties in interoperating with Windows

[Alan Cox]

> and hire all the lawyers they'd want...at the end of the day, could you
> just go to court and say:

Oh glorious naivety.

> The law seems pretty damn clear on this issue (in fact Subsection 3.c
> sounds like I could quite happily send it to someone in the USA if they
> can do the work better than I).

You release the code

The next day an injunction arrives in the post

The day after that the large corporate lawyers slap a secrecy order on you
preventing you discussing the case with anyone but your legal representative
by convincing some pissed geriatric moron in the high court that if he
doesnt sign it you'll tell the whole world and harm them irrepairably

Over the next week 10-15,000 pages of documents arrive for you, followed
by a couple of thousand discovery requests.

A week later while you are reading them they file for summary judgement 
because you have failed to file the defence. You lose. Game over

Then the court costs bankrupt you.


Sound like fiction - it happens _regularly_. Its very rare (McLibel case
is extremely unusual) that it can ever be talked about on a per case basis,
or that the good guys win.

This is why everyone doing innovative computing work does it outside of the
US, and outside of most of the EU. 

Alan

Re: Difficulties in interoperating with Windows

[Helge Hafting]

Alan Cox wrote:

> Over the next week 10-15,000 pages of documents arrive for you, followed
> by a couple of thousand discovery requests.
> 
> A week later while you are reading them they file for summary judgement
> because you have failed to file the defence. You lose. Game over

Ah, a DOS attack in another form.

Helge Hafting

Re: Difficulties in interoperating with Windows

[David Schwartz]

>Reverse engineering is taking the published specifications, creating
>software that should function in an equivalent manner. Testing it is
>difficult since you have to be careful not to import patented/copyrighted
>algorithms. You have to compair the inputs/outputs for the software with the
>inputs/outputs of the original.

	No, developing a product to meet given specifications is the ordinary sort 
of forward engineering. Reverse engineering is when you attempt to determine 
the specifications by looking at, taking apart, or probing the final product.

	DS

Re: Difficulties in interoperating with Windows

[Anton Altaparmakov]

At 15:06 09/01/02, Jesse Pollard wrote:
> > IANAL, but I wondered why there is such difficulty with making Linux work
> > completely with Windows systems?  I guess part of it may be that Windows
> > is closed source but as reverse engineering for interoperability is legal
> > in the UK (regardless of what the End User License states), is the problem
> > that it is difficult to read the Assembly easily?
>
>That is not reverse engineering - (at least not MY understanding) - you are
>re-translating a copyrighted work.

Actually disassembly is the first and very important step in reverse 
engineering (tha is in my understanding and that is how the law sees it as 
far as I understand EU law, note IANAL... so don't take my word for it).

>If the translation back into the binary form creates the same binary then 
>you have an exact translation. You also have a lawsuit pending. Otherwise 
>you could just disassemble the entire windows OS, claim it as your 
>"re-engineered source", and sell/publish it.

You couldn't do that, no.

>This is not legal in most locations.
>
>Reverse engineering is taking the published specifications, creating 
>software that should function in an equivalent manner.

That is not true. True reverse engineering is disassembling the binary, 
determining the specifications laid down in the binary, then writing a 
program to implement those specifications (again IANAL and your legislation 
may well vary)

You are spoilt if you have the specifications already available... With 
NTFS I have spent months going through the ntfs driver disassembly in 
IDAPro (www.datarescue.com), the best disassembler there is AFAIK, 
determining how things work.

>Testing it is difficult since you have to be careful not to import 
>patented/copyrighted algorithms. You have to compair the inputs/outputs 
>for the software with the inputs/outputs of the original.

Er, you have to have the same algorithms or at least you need to achieve 
the same input and output which often requires the exact same algorithm 
otherwise you cannot achieve the same input/output...

To give a concrete example from ntfs, when collating attribute names (and 
file names for the matter) in order to determine where to place them in an 
inode, if you do not apply all collation criteria found in the windows 
driver, you will inevitably do the collation wrong at least in some corner 
cases and you have a broken filesystem on your hands when you are writing.

>The end result is usable. Linux itself can be considered a reversed 
>engineered UNIX.

I would disagree. You have all the specifications defined in all the 
standards and you are just implementing them. That has nothing to do with 
reverse engineering. That is just implementing a specification/design, that 
is normal everyday programming. Parts of Linux are reverse engineered, sure 
thing, but not the whole OS.

>At least a sufficient amount of the specifications are available.
>
>The problem with most M$ software is that the published specifications are 
>not complete, access to the inputs are not always available (it is ALSO 
>covered by the proprietary/trade secrets/other restrictions). Sometimes 
>the output is not available (at least in some countries - DMCA again).
>
>This is also the problem with some device interfaces. If the company claims
>"trade secret" or "propriatary information" then the published 
>pecifications will not be usable when reverse engineering device drivers.

You can use whatever information you like as long as you are not disallowed 
to do so by your local legislation. The EU is very liberal with regards to 
reverse engineering and the US is I gather a forbidding country.

>This problem has been attacked in the past as "it SHOULD work like this, 
>if I can find a register that looks like what I expect, then it MIGHT 
>work"... Some IDE controllers were originally developed this way - at 
>least one video board was done this way. (I think even a PCI bridge was 
>deciphered this way)
>
> > Is there not a project on Linux to convert assembly back to C?   Would
> > this be exceptionally hard?
>
>Not hard - just illegal when using it to disassemble proprietary software.
>Debuggers do this very frequently, to the point that I would say "all the
>time" except for debuggers of interpreted languages.

It is not only exceptionally hard, it completely impossible to do 
automatically. Assembler code cannot be translated to meaningful C code 
without human intervention. Of course if you have full symbolic 
information, it can be done, but no commercial software includes symbolic 
information if the company has any sense. If you were to write an asm -> C 
converter the only thing it could ever do is to name all functions and 
variables func1, func2, ... and var1, var2, etc without full type 
information, at least not C types, no names and no meanings. (That is what 
IDAPro does in fact but it keeps the code in assembler and doesn't convert 
it to C.)

The real hard core part of reverse engineering is to understand what the 
heck the functions do, why they do it, what it means and to deduce the 
specifications from that. (With ntfs we do have partial debugging symbols, 
for all functions ntfs exports to other parts of windows, and we have some 
of the function names, but by no means all.)

Comparing input/output is also very valuable in the course of reverse 
engineering but by no means sufficient because you will (almost) never find 
out all corner/special cases that way. Depending on what you are reverse 
engineering it may be sufficient but it is extremely unlikely to give you 
the full specifications.

> > Just wondered why Linux has struggled for a while to interoperate with
> > Windows completely...

Because they don't give us specifications, because they don't give us the 
source code either, and because in many countries people are not allowed to 
reverse engineer the binaries either. And even that is illegal in some 
countries. Which doesn't leave you anything much to work from...

Luckily in the EU we can do just about anything with respect to reverse 
engineering as long as the purpose is interoperability which covers ntfs 
nicely. The myth of the US being a very free country is only a myth after 
all. The EU is much freer. (-;

Best regards,

         Anton

DISCLAIMER: Parts of this email are to be taken with a pinch of salt... 
Especially any parts describing laws shouldn't be relied on as IANAL...


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: Difficulties in interoperating with Windows

[Jesse Pollard]

...
> Er, you have to have the same algorithms or at least you need to achieve 
> the same input and output which often requires the exact same algorithm 
> otherwise you cannot achieve the same input/output...
>

Yup - and then you hit the "trade secrets" problem.

> To give a concrete example from ntfs, when collating attribute names (and 
> file names for the matter) in order to determine where to place them in an 
> inode, if you do not apply all collation criteria found in the windows 
> driver, you will inevitably do the collation wrong at least in some corner 
> cases and you have a broken filesystem on your hands when you are writing.

I believe the collating sequence/filenames is documentd. What isn't documented
is how the journal file is handled. How recovery is handled.

I think trying to make that compatable hits the trade secrets. Compatability
is needed if you expect to take a partition from one OS to another and still
have the possible crash conditions handled. NTFS write was (briefly) available
until the lawyers came to the door. Along with an external tool to recover
NTFS file systems.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Difficulties in interoperating with Windows

[Alan Cox]

> > the same input and output which often requires the exact same algorithm 
> > otherwise you cannot achieve the same input/output...
> 
> Yup - and then you hit the "trade secrets" problem.

In the EU generally not. For hardware the normal approach is
to use patented interfaces to ensure for example that nobody can sell
replacement ink cartridges for your printer. For software its trickier
and corporations are still working on schemes

Trade secrets really only become a problem if you've seen the original for
other reasons. A microsoft employee leaving MS who spends the next month
writing an ntfs for unix is likely to get into some trouble 8)

Re: Difficulties in interoperating with Windows

[Anton Altaparmakov]

At 16:48 09/01/02, Jesse Pollard wrote:
> > To give a concrete example from ntfs, when collating attribute names (and
> > file names for the matter) in order to determine where to place them in an
> > inode, if you do not apply all collation criteria found in the windows
> > driver, you will inevitably do the collation wrong at least in some corner
> > cases and you have a broken filesystem on your hands when you are writing.
>
>I believe the collating sequence/filenames is documentd.

I tell you it isn't. The documentation you can find in layout.h in 
ntfs-driver-tng and linux-ntfs and in the tng source is written by _me_ 
based on reverse engineering the ntfs driver. It took me several months to 
do this. If you look at the collation code and particular the name 
collation you will see why... And no this is NOT documented anywhere AFAIK.

>What isn't documented is how the journal file is handled.

Yes, that isn't documented either. And according to Jeff Merkey there are 
patent infringement issues with the log file.

>How recovery is handled.

It's documented on a concept level in "Inside Windows NT/2000" 2nd/3rd Ed 
from Microsoft press. But that doesn't help zilch in understanding the 
on-disk layout.

>I think trying to make that compatable hits the trade secrets. Compatability
>is needed if you expect to take a partition from one OS to another and still
>have the possible crash conditions handled.

Yes. ntfs tng will just unconditionally empty the journal completely as 
soon as someone performs a write mount. Currently you need to run ntfsfix 
(from linux-ntfs package) in order to achieve this after writing with the 
current ntfs driver.

>NTFS write was (briefly) available until the lawyers came to the door.

This is simply not true. The Linux driver is able to write. The problem is 
it does so very badly and corrupts the partition which doesn't make it very 
useful. At present I have stable write support to normal, existing files 
but directory operations are still broken. However I have now abandoned 
trying to fix the old driver's write support. NTFS TNG will have full write 
support eventually (but read support is not quite finished yet...).

>Along with an external tool to recover NTFS file systems.

If you are refering to my ntfsfix that is available from sourceforge, 
linux-ntfs project and package of same name.

If you are refering to microsoft's diskedit that is available on the 
Windows NT 4 SP4 CDROM which one can obtain from various sources.

No other recovery tool has been made publically available AFAIK.

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/