Indy: ToBytes and BytesToString

Hi,

I am using D2009 Enterprise, and I found the next problem:


procedure SendCommBlock(IO: TIdIOHandler; const sBuf: AnsiString);
var
   s: AnsiString;
   Buf: TIdBytes;
begin
  Buf := ToBytes(sBuf, en8Bit);
  s := BytesToString(Buf, en8Bit);

If the original sBuf has into it any characters from #0 to #255, then the 
resulting s can be different from the original. Sounds as something is 
changing when encoding/decoding.

I tried with the defaul en7Bit, but same problem.

What is missed?

Thank you.

Adriano.
0
Adriano
11/8/2011 2:46:54 PM
embarcadero.delphi.tools 5366 articles. 2 followers. Follow

3 Replies
6050 Views

Similar Articles

[PageSpeed] 17

"Adriano Macome" <adrianomacome@hotmail.com> wrote in message 
news:417828@forums.embarcadero.com...

>  Buf := ToBytes(sBuf, en8Bit);

In D2009+, ToBytes() does not have an overload available for AnsiString 
values, so the RTL is implicitally converting your AnsiString to a 
UnicodeString before ToBytes() sees it.  Ansi characters #128 - #255 are 
subject to codepage interpretation, and the RTL uses the OS default Ansi 
codepage when converting an AnsiString to a UnicodeString (if you change 
your function to accept a RawByteString instead of an AnsiString, then the 
conversion will use whatever codepage the RawByteString is holding at the 
time the function is called).

To put the raw data of a AnsiString (since you are using en8Bit) into a 
TIdBytes without any conversion, use RawToBytes() instead:

{code:delphi}
if sBuf <> '' then
  Buf := RawToBytes(sBuf[1], Length(sBuf))
else
  Buf := nil;
{code}

Or, use RawToBytesF():

{code:delphi}
SetLength(Buf, Length(sBuf));
if sBuf <> '' then RawToBytesF(Buf, sBuf[1], Length(sBuf));
{code}

>  s := BytesToString(Buf, en8Bit);

Likewise, there is no overload of BytesToString() available for AnsiString 
in D2009+, either.  Assuming the TIdBytes is filled with Ansi data, you are 
going to end up with a UnicodeString that contains Unicode characters 
U+0000 - U+00FF.  When converting a UnicodeString to an AnsiString, the OS 
default Ansi codepage is used again, and there is no guarantee that Unicode 
characters U+0080 - U+00FF will map 1-to-1 to Ansi characters #128 - #255. 
It depends on which codepage is used for the conversion and which characters 
are present, so there is still potential for data loss.

To put the raw data of a TIdBytes (since you are using en8Bit again) into an 
AnsiString without any conversion, use BytesToRaw() instead:

{code:delphi}
SetLength(s, Length(Buf));
if s <> '' then BytesToRaw(Buf, s[1], Length(Buf));
{code}

> If the original sBuf has into it any characters from #0 to #255, then the
> resulting s can be different from the original.

Yes, because of the implicit Ansi<->Unicode conversions that are occuring 
inside the RTL outside of Indy.  ANY time an Ansi<->Unicode conversion 
occurs for Ansi characters > #127, data loss MAY occur.  Ansi characters 
#00 - #127 are always safe, as they represent the ASCII charset, and map 
1-to-1 to the same values in all Ansi charsets and Unicode.

> Sounds as something is changing when encoding/decoding.

Yes, it is.

> I tried with the defaul en7Bit, but same problem.

en7Bit is even worse, because it does not even support ASCII characters 
#00 - #127, not Ansi characters > #127 at all.

-- 
Remy Lebeau (TeamB)
0
Remy
11/8/2011 8:21:41 PM
Hi Remy;

Your answer is amazing, thank you very much. I tried your suggestions and 
they work fine, nice. Also, I understand all your clear explanations about 
Ansi-Unicode conversion, nice!.

Best regards,

Adriano.


"Remy Lebeau (TeamB)" <no.spam@no.spam.com> escribió en el mensaje de 
noticias news:417967@forums.embarcadero.com...
> "Adriano Macome" <adrianomacome@hotmail.com> wrote in message 
> news:417828@forums.embarcadero.com...
>
>>  Buf := ToBytes(sBuf, en8Bit);
>
> In D2009+, ToBytes() does not have an overload available for AnsiString 
> values, so the RTL is implicitally converting your AnsiString to a 
> UnicodeString before ToBytes() sees it.  Ansi characters #128 - #255 are 
> subject to codepage interpretation, and the RTL uses the OS default Ansi 
> codepage when converting an AnsiString to a UnicodeString (if you change 
> your function to accept a RawByteString instead of an AnsiString, then the 
> conversion will use whatever codepage the RawByteString is holding at the 
> time the function is called).
>
> To put the raw data of a AnsiString (since you are using en8Bit) into a 
> TIdBytes without any conversion, use RawToBytes() instead:
>
> {code:delphi}
> if sBuf <> '' then
>  Buf := RawToBytes(sBuf[1], Length(sBuf))
> else
>  Buf := nil;
> {code}
>
> Or, use RawToBytesF():
>
> {code:delphi}
> SetLength(Buf, Length(sBuf));
> if sBuf <> '' then RawToBytesF(Buf, sBuf[1], Length(sBuf));
> {code}
>
>>  s := BytesToString(Buf, en8Bit);
>
> Likewise, there is no overload of BytesToString() available for AnsiString 
> in D2009+, either.  Assuming the TIdBytes is filled with Ansi data, you 
> are going to end up with a UnicodeString that contains Unicode characters 
> U+0000 - U+00FF.  When converting a UnicodeString to an AnsiString, the OS 
> default Ansi codepage is used again, and there is no guarantee that 
> Unicode characters U+0080 - U+00FF will map 1-to-1 to Ansi characters 
> #128 - #255. It depends on which codepage is used for the conversion and 
> which characters are present, so there is still potential for data loss.
>
> To put the raw data of a TIdBytes (since you are using en8Bit again) into 
> an AnsiString without any conversion, use BytesToRaw() instead:
>
> {code:delphi}
> SetLength(s, Length(Buf));
> if s <> '' then BytesToRaw(Buf, s[1], Length(Buf));
> {code}
>
>> If the original sBuf has into it any characters from #0 to #255, then the
>> resulting s can be different from the original.
>
> Yes, because of the implicit Ansi<->Unicode conversions that are occuring 
> inside the RTL outside of Indy.  ANY time an Ansi<->Unicode conversion 
> occurs for Ansi characters > #127, data loss MAY occur.  Ansi characters 
> #00 - #127 are always safe, as they represent the ASCII charset, and map 
> 1-to-1 to the same values in all Ansi charsets and Unicode.
>
>> Sounds as something is changing when encoding/decoding.
>
> Yes, it is.
>
>> I tried with the defaul en7Bit, but same problem.
>
> en7Bit is even worse, because it does not even support ASCII characters 
> #00 - #127, not Ansi characters > #127 at all.
>
> -- 
> Remy Lebeau (TeamB)
0
Adriano
11/9/2011 12:13:13 PM
"Remy Lebeau (TeamB)" <no.spam@no.spam.com> wrote in message 
news:417967@forums.embarcadero.com...

> en7Bit is even worse, because it does not even support ASCII
> characters #00 - #127, not Ansi characters > #127 at all.

I meant: "because it *only* supports ASCII ...".

-- 
Remy Lebeau (TeamB)
0
Remy
11/9/2011 6:47:38 PM
Reply:

Similar Artilces:

Tools, Tools, Tools
</quot> Couple useful things for IT admins out there. I've packaged up Werner and Feder's PoC scanner via py2exe here. You can now simply run: </end_quot> http://www.doxpara.com/ -- "Never drive faster than your ANGEL can fly" ...

Converting Delphi 2007 Indy 10.2.3 to Delphi 2009 Indy 10.5.5 [Edit]
Hello, I am currently attempting to port over a Delphi 2007 project that uses Indy 10.2.3 (very successfully) to Delphi 2009 and Indy 10.5.5 (I just got the latest development build this morning). I think I am running into an encoding issue, but am not sure. Specifically, IDHTTP with SSL calls an old CGI and the CGI returns a .zip file and I then save it to the disk. In 2007 and before this worked perfectly. In 2009, it is not. Here is the examples of the 2 different results (though cut way short in the post) I am getting back: 2007: 'PK'#3#4#$14#0#0#0#8#0'rLQ9žrPb€'#0...

delphi 7 Indy 9 and Indy 10
Hi, I can upgrade to indy 10 in delphi 7. But I have discover that indy 9 and Indy 10 have some different properties, so i had to change my old programs that were made in indy 9 to upgrade it to new version, but it is a long work. So I want to know if it is possible to install in the same delphi 7 both versions indy 10 and indy 9. I have tried to do it but i received a error message: Cannot load package 'IndySystem70'. It contains unit 'IdWinSock2', which is also contained in package 'Indy70'. Some can give me ideas or a link to read how to install both versi...

ANN: wxForms for Delphi
We are pleased to announce the release of our wxWidgets Form Designer for Delphi - wxForms for Delphi 1.1. wxForms for Delphi is an integrated form designer plugin for Borland /CodeGear Delphi that helps to create cross platform applications for Windows, Mac OSX and Linux using single source base. The wxForms code from Delphi can be compiled in Mac OSX and Linux without much change using FreePascal. wxForms for Delphi allows you to use all the Delphi's powerful Form Designer/ Editor features (Form Alignment, Object inspector, Component Palette etc ) to quickly create Cross platfor...

Indy IdHMACSHA1 different results in Delphi 2007 and Delphi 2009
Hello newsgroup A test of the IdHMACSHA1 encryption method shows different results, and I am not sure if there is an error in my code or in the Indy code shipped with Delphi 2009. The Delphi 2007 version works in an application which communicates with Amazon Web Services and so it seems to be the correct one. Any suggestions are very welcome, maybe I just need to download a newer version of Indy? program Test; {$APPTYPE CONSOLE} uses IdHMACSHA1, IdCoderMIME, IdGlobal, SysUtils; function Base64Encode(const Input: string): string; begin Result := TIdEncode...

Indy E-Mail Problem (Indy 10, Delphi 2007)
Hello, we use Indy 10 with Delphi 2007 to get E-Mail using IMAP. We recieved an E-Mail which is not handled correctly. The E-Mail looks like this {code} Return-Path: <test@mail.com>; From: "Test Tester" <test@mail.com>; To: <test@mail.com>; Subject: Order Date: Wed, 7 Mar 2012 16:50:40 +0100 Message-ID: <15D526BEB8091D43859549D9E16E370FBA3123> MIME-Version: 1.0 Content-Type: text/html; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQFk6K6fcF3...

migrating from Delphi 6 With Indy 10 to XE7 with Indy 10
I updated the original Indy in D6 to version 10 several years ago. Now I want to migrate my application from D6 to XE7 and would like some feedback on the best route to take. I usually send data using readln and writeln statements. The data is typically XML format. Since migrating to XE7 will include potential unicode data what is the best approach to take when reading and writing data? Will writeln and readln work in these cases or should I be using a different strategy to send unicode data between the tidtcpclient and tidtcpserver applications? al wrote: > I usually send data ...

looking for other tools then delphi
Hi: I have been using delphi from version 2. With the time, I feel delphi's power is diminishing. Personally I think delphi doesn't give me surprise after version 5. All tcp/ip, internet components are from third party which is buggy and hardly to build web application, report tools are third party which is buggy too, and it's even impossible to build application on portable device by using delphi. Very disappointing. I feel guilty and a bit embaraccement to your guys to ask which other tools are better if I switch off delphi. But it's boss intention especiall...

Delphi 2009 and Indy
I installed Delphi 2009. As far as I can see Indy 10 is installed with it. Now I try to get a program working that I made under Delphi 7. It uses TIDHashMessageDigest. Now I cannot use it because of a undeclared identifier. I added IDGlobal, IDHash, IDHashMessageDigest to the uses list, but no result. What went wrong, maybe something with installation? Please help. Franklin Bockstael wrote: > > What went wrong, maybe something with installation? No, I suspect that it has something to do with Indy. I'd suggest asking in one of the technical groups that covers ...

Indy support and Delphi
Hi all, I've been off the support site for nearly a year since the move to Embarcadero. Does anyone know where to get help with an Indy/Delphi 5 issue ? I was ready to pay the Indy team to help but they have shut down support ... wont even do it for money. I have an issue with the POP3 server that is showing up and need some short technical advise. TIA, Del Delbert Murray schrieb: > Hi all, > I've been off the support site for nearly a year since the move to > Embarcadero. Does anyone know where to get help with an Indy/Delphi 5 issue > ? I was ready to pay ...

Delphi 7, Indy
I am trying to learn to use the Indy components with D7. I'm using the book "Mastering Delphi 7". I can't follow his examples, and below is a couple of excerpts that I need help with. The first relates to a form with a TIdTCPClient component on it. He indicates the following: object IdTCPServer1: TIdTCPServer defaultPort = 1050 end Is he indicating that the property should be set to 1050, or is this some type of code? If so, I'm lost. I would appreciate some help Another puzzler is object IdTCPServer1: TIdTCPServer CommandHandler...

Delphi Build tools
Similar to my last question about DUnit, what about Want? Is Want what I want when it comes to a build too, or has Want been found wanting (at least in comparison to something else)? What about the SmartBear/AutomatedQA offerings in these areas? Are they viable/worth the effort to learn/worth the $? Again, I need comments/suggestions relative to Delphi 7 only. On 2012-01-30 12:16:36 -0500, Clay Shannon <> said: > What about the SmartBear/AutomatedQA offerings in these areas? Are they > viable/worth the effort to learn/worth the $? Not sure if those are the only one&...

Delphi is good tool too.
Of course,PB is a very good tool especial for database programing .But I think its not fit to program system programs such as DLL,create ActiveX control,etc.While Delphi is a very smart tool,It is a true program tool,it has a very fast compiler,it can build true native code exe.It can create DLL,Active X,web server application.Though it does not have Datawindow,it has a lot of data controls which you can build what thing you need and the third party invontor have develop powerful datacontrol like Datawindow.And i think Delphi 's IDE is more efficent than PB's with is code insig...

Delphi Tools -Configure
Hi All, Can we have an option under Delphi Tools=>Configure Tools to include "run as administrator" please? regards SeaCay > {quote:title=Colin Kemp wrote:}{quote} > Hi All, > > Can we have an option under Delphi Tools=>Configure Tools to include > "run as administrator" please? You should add a feature request in QC. In the mean time, you can build your own. In his forst screencast, Jim McKeeth demonstrates how to create a new Tools menu item to open the current file from the IDE in Windows Explorer. You could do something si...

Web resources about - Indy: ToBytes and BytesToString - embarcadero.delphi.tools

Checked exceptions I love you, but you have to go
Once upon a time Java created an experiment called checked-exceptions, you know, you have to declare exceptions or catch them. Since that time, ...

Resources last updated: 12/31/2015 4:11:00 PM