From MAILER-DAEMON Mon Jul 16 07:31:01 2012
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1SqjW1-0007s8-F0
	for mharc-bug-diffutils@gnu.org; Mon, 16 Jul 2012 07:31:01 -0400
Received: from eggs.gnu.org ([208.118.235.92]:56072)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sanvila@unex.es>) id 1SqjVu-0007pw-En
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 07:30:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sanvila@unex.es>) id 1SqjVt-0006Xn-9H
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 07:30:54 -0400
Received: from pizarro.unex.es ([158.49.8.2]:57104)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sanvila@unex.es>) id 1SqjVt-0006Xh-2K
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 07:30:53 -0400
Received: from guadiana.unex.es (guadiana.unex.es [158.49.17.23])
	by pizarro.unex.es (Postfix) with ESMTP id 9FEDC14810D;
	Mon, 16 Jul 2012 13:30:50 +0200 (CEST)
Received: from cantor.unex.es (cantor.unex.es [158.49.50.20])
	by guadiana.unex.es (Postfix) with ESMTPS id A47094B96B;
	Mon, 16 Jul 2012 13:30:48 +0200 (CEST)
Date: Mon, 16 Jul 2012 13:30:48 +0200 (CEST)
From: Santiago Vila <sanvila@unex.es>
To: bug-diffutils@gnu.org
Message-ID: <alpine.DEB.2.02.1207161227350.11194@cantor.unex.es>
User-Agent: Alpine 2.02 (DEB 1266 2009-07-14)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 158.49.8.2
Cc: 680990-forwarded@bugs.debian.org, jmg <jeanmichel.123@free.fr>
Subject: [bug-diffutils] Bug#680990: Diff does not show BOM difference. (fwd)
X-BeenThere: bug-diffutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "All diffutils discussion." <bug-diffutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-diffutils>,
	<mailto:bug-diffutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/bug-diffutils>
List-Post: <mailto:bug-diffutils@gnu.org>
List-Help: <mailto:bug-diffutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-diffutils>,
	<mailto:bug-diffutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 16 Jul 2012 11:31:00 -0000

Hello.

I received this report from the Debian bug system.
The full log for this bug may be found here:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=680990

but it includes a long discussion between the submitter and myself
which I'll try to summarize here for brevity.

In the first place, the submitter complains that whenever two files
are diffed, one containing a BOM and another one not containing a BOM,
the BOM itself is not clearly visible in the diff, which might make
the user a little bit puzzled.

My reply to this is that the BOM is invisible, like spaces, and diff
is not to blame for that.

The submitter then complains that if BOM is not visible but a text file
with BOM is "equivalent" to the same text without BOM, then it would
make sense for "diff -w" to discard differences based on BOM presence.

Another suggestion from the submitter is that diff could show a message
similar to "\ No newline at end of file" when the difference is based
on presence/lack of BOM.

Thanks.


From MAILER-DAEMON Mon Jul 16 10:25:47 2012
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1SqmF9-0005ph-Jy
	for mharc-bug-diffutils@gnu.org; Mon, 16 Jul 2012 10:25:47 -0400
Received: from eggs.gnu.org ([208.118.235.92]:45336)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eggert@cs.ucla.edu>) id 1SqmF2-0005ok-LU
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 10:25:46 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eggert@cs.ucla.edu>) id 1SqmEs-0003Tt-B8
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 10:25:40 -0400
Received: from smtp.cs.ucla.edu ([131.179.128.62]:50697)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eggert@cs.ucla.edu>) id 1SqmEs-0003Tg-4q
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 10:25:30 -0400
Received: from localhost (localhost.localdomain [127.0.0.1])
	by smtp.cs.ucla.edu (Postfix) with ESMTP id 775CAA60004;
	Mon, 16 Jul 2012 07:25:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu
Received: from smtp.cs.ucla.edu ([127.0.0.1])
	by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 1AjEpC+zCgrG; Mon, 16 Jul 2012 07:25:29 -0700 (PDT)
Received: from [192.168.1.4] (pool-108-23-119-2.lsanca.fios.verizon.net
	[108.23.119.2])
	by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 02E6BA60001;
	Mon, 16 Jul 2012 07:25:28 -0700 (PDT)
Message-ID: <50042458.1070308@cs.ucla.edu>
Date: Mon, 16 Jul 2012 07:25:28 -0700
From: Paul Eggert <eggert@cs.ucla.edu>
Organization: UCLA Computer Science Department
User-Agent: Mozilla/5.0 (X11; Linux i686;
	rv:13.0) Gecko/20120615 Thunderbird/13.0.1
MIME-Version: 1.0
To: Santiago Vila <sanvila@unex.es>
References: <alpine.DEB.2.02.1207161227350.11194@cantor.unex.es>
In-Reply-To: <alpine.DEB.2.02.1207161227350.11194@cantor.unex.es>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 131.179.128.62
Cc: bug-diffutils@gnu.org, 680990-forwarded@bugs.debian.org,
	jmg <jeanmichel.123@free.fr>
Subject: Re: [bug-diffutils] Bug#680990: Diff does not show BOM difference.
 (fwd)
X-BeenThere: bug-diffutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "All diffutils discussion." <bug-diffutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-diffutils>,
	<mailto:bug-diffutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/bug-diffutils>
List-Post: <mailto:bug-diffutils@gnu.org>
List-Help: <mailto:bug-diffutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-diffutils>,
	<mailto:bug-diffutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 16 Jul 2012 14:25:46 -0000

It'd be reasonable to have 'diff' ignore byte-order-marks,
just as it already ignores things like white space,
when given a new option to do that.

Someone would have to write the code and documentation,
though.


From MAILER-DAEMON Mon Jul 16 21:18:30 2012
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1SqwQo-0007C2-OV
	for mharc-bug-diffutils@gnu.org; Mon, 16 Jul 2012 21:18:30 -0400
Received: from eggs.gnu.org ([208.118.235.92]:49881)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeanmichel.123@free.fr>) id 1SqvaE-0001h1-5q
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 20:24:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jeanmichel.123@free.fr>) id 1SqvaD-00023v-1D
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 20:24:10 -0400
Received: from smtp5-g21.free.fr ([212.27.42.5]:37773)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeanmichel.123@free.fr>) id 1SqvaC-00023N-DQ
	for bug-diffutils@gnu.org; Mon, 16 Jul 2012 20:24:08 -0400
Received: from zimbra28-e5.priv.proxad.net (unknown [172.20.243.178])
	by smtp5-g21.free.fr (Postfix) with ESMTP id 5E582D48035;
	Tue, 17 Jul 2012 02:24:00 +0200 (CEST)
Date: Tue, 17 Jul 2012 02:23:59 +0200 (CEST)
From: jeanmichel.123@free.fr
To: Paul Eggert <eggert@cs.ucla.edu>
Message-ID: <954401472.22258888.1342484639988.JavaMail.root@zimbra28-e5.priv.proxad.net>
In-Reply-To: <50042458.1070308@cs.ucla.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Originating-IP: [84.97.14.22]
X-Mailer: Zimbra 7.2.0-GA2598 (ZimbraWebClient - FF3.0 (Linux)/7.2.0-GA2598)
X-Authenticated-User: jeanmichel.123@free.fr
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 212.27.42.5
X-Mailman-Approved-At: Mon, 16 Jul 2012 21:18:29 -0400
Cc: bug-diffutils@gnu.org, 680990-forwarded@bugs.debian.org,
	Santiago Vila <sanvila@unex.es>
Subject: Re: [bug-diffutils] Bug#680990: Diff does not show BOM difference.
 (fwd)
X-BeenThere: bug-diffutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "All diffutils discussion." <bug-diffutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/bug-diffutils>,
	<mailto:bug-diffutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/bug-diffutils>
List-Post: <mailto:bug-diffutils@gnu.org>
List-Help: <mailto:bug-diffutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/bug-diffutils>,
	<mailto:bug-diffutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 17 Jul 2012 00:24:11 -0000

I did not integrated it in diff, nor tested it.
Because I do not know diff code and compilation enough.

But as it appears there is no unicode support in diff tool, such a code that IGNORE_ALL_UNICODE_SPACE in complement of IGNORE_ALL_SPACE
for instance:
       toUTF8_1 ( 0x0009 ),
       toUTF8_1 ( 0x000A ),
       toUTF8_1 ( 0x000B ),
       toUTF8_1 ( 0x000C ),
       toUTF8_1 ( 0x000D ),
       toUTF8_1 ( 0x0020 ),
// c2
       toUTF8_2 ( 0x0085 ),
       toUTF8_2 ( 0x00A0 ),
// e1
       toUTF8_3 ( 0x1680 ),
       toUTF8_3 ( 0x180E ),
// e2 80
       toUTF8_3 ( 0x2000 ),
       toUTF8_3 ( 0x2001 ),
       toUTF8_3 ( 0x2002  ),
       toUTF8_3 ( 0x2003  ),
       toUTF8_3 ( 0x2004  ),
       toUTF8_3 ( 0x2005  ),
       toUTF8_3 ( 0x2006  ),
       toUTF8_3 ( 0x2007  ),
       toUTF8_3 ( 0x2008  ),
       toUTF8_3 ( 0x2009  ),
       toUTF8_3 ( 0x200A  ),
       toUTF8_3 ( 0x2028  ),
       toUTF8_3 ( 0x2029  ),
       toUTF8_3 ( 0x202F  ),
// e2 81
       toUTF8_3 ( 0x205F  ),
// e3
       toUTF8_3 ( 0x3000  ),
// ef
       toUTF8_3 ( 0xfeff  ),
(taken from http://en.wikipedia.org/wiki/Whitespace_character and added BOM)
might look like:


//////////////////////////////////////////////////////
typedef unsigned char byte;
/** Check if this byteSequence start with an utf8 space character */
// Return the number of bytes which match the space character if any.
int isUtf8Space(byte*input)
{
  switch (input[0])
  {
    case 0x0009:
    case 0x000A:
    case 0x000B:
    case 0x000C:
    case 0x000D:
    case 0x0020:
      return 1;
      break;
    case 0x00c2:
      if ( (input[1]==0x00a0) || (input[1]==0x0085) )
        return 2;
      break;
    case 0x00e1:
      if (( input[1]==0x009a) && (input[2]==0x0080) )
        return 3;
      if (( input[1]==0x00a0) && (input[2]==0x008e) )
        return 3;
      break;
    case 0x00e2:
      switch (input[1])
        {  
           case 0x0080:
             if ( ( input[2]>=0x80) &&
                   (input[2]<=0x8a)   )
                 return 3;
             if ( input[2]==0xa8)
                 return 3;
             if ( input[2]==0xa9)
                 return 3;
             if ( input[2]==0xaf)
                 return 3;
            break;
           case 0x0081:
             if ( input[2]==0x9f)
                 return 3;
            break;
           case 0x0097:
             if ( input[2]==0xbf)
                 return 3;
            break;
        }
      break;
    case 0x00e3:
      if ( (input[1]==0x0080) && (input[2]==0x0080) )
        return 3;
      break;
    case 0x00ef:
      if ( (input[1]==0x00bb) && (input[2]==0x00bf) )
        return 3;
      break;
  }
  return 0;
}

/** test file*/
int  main(int argc, char**argv)
{
  char * t1 = argv[1];
  char * t2 = argv[2];
  int n;
  char c1, c2;
            //case IGNORE_ALL_UNICODE_SPACE:
              /* For -w, just skip past any white space.  */
              while ( (n=isUtf8Space (t1)) && *t1 != '\n')  t1+=n;
              while ( (n=isUtf8Space (t2)) && *t2 != '\n')  t2+=n;
                  c1 = *t1;
                  c2 = *t2;
              //break;
  printf ("<%s\n", t1);
  printf (">%s\n", t2);
  return strcmp(t1, t2);
}
//////////////////////////////////////////////////////

Unfortunately, it only handle UTF-8 and not UTF-16.


----- Mail original -----

It'd be reasonable to have 'diff' ignore byte-order-marks,
just as it already ignores things like white space,
when given a new option to do that.

Someone would have to write the code and documentation,
though.


