Delphi XE broken UTF8

Hi, I am using the Rad Studio XE trial edition to see how hard an upgrade of our project will be.

Getting to compile and link was pretty easy.

However, the UnicodeToUtf8 function appears to be badly broken making our app unusable. Please can someone with the full release confirm this and maybe go into the System source file (not available in trial version) to analyse?

It is easily reproducable (and this code works fine on D2010). The function appears to be not null terminating (as the help promises it will):

{code}

function UTF8Str(const WS: string): AnsiString;
var
  szTitle: array[0..511] of AnsiChar;
begin
  szTitle[0] := #0;
  szTitle[3] := #1;
  UnicodeToUtf8(szTitle, 511, PChar(WS), Length(WS));
  Result := szTitle;
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  ShowMessage(UTF8Str('Man'));
end;
{code}

NOTE: I don't normally have the #1 line but this highlights the memory is just left with possible junk in at this location.

I can rewrite my UTF8Str to null terminate explicitly but I am worried about other breakages...

Cheers
Dave
0
Dave
9/1/2010 10:35:34 AM
embarcadero.delphi.general 4258 articles. 0 followers. Follow

10 Replies
2250 Views

Similar Articles

[PageSpeed] 36

Am 01.09.2010 12:35, schrieb Dave Speight:
> Please can someone with the full release confirm this and maybe go into the System source file (not available in trial version) to analyse?

The procedure UnicodeToUtf8 has been modified in XE to prepare for 
cross-platform. Seems like the null terminator check has fallen through. 
You should file a QC for this.

Uwe Raabe
-- 
Uwe's Blog: The Art of Delphi Programming <http://www.uweraabe.de/>
0
Uwe
9/1/2010 11:01:46 AM
Dave Speight wrote:

> Please can someone with the full release confirm this and maybe go
> into the System source file (not available in trial version) to
> analyse?

See for the "why" Uwe's reply.

Workarounds:
use
  UnicodeToUtf8(szTitle, 511, PChar(WS), Cardinal(-1));
or
  Result := UTF8Encode(WS).

-- 
Pieter

"If it weren't for electricity we'd all be watching television by
 candlelight." -- George Gobel.
0
Pieter
9/1/2010 12:25:49 PM
> {quote:title=Uwe Raabe wrote:}{quote}
> The procedure UnicodeToUtf8 has been modified in XE to prepare for 
> cross-platform. Seems like the null terminator check has fallen through. 
> You should file a QC for this.

That's why I've implemented our own versions of UTF-8/Unicode/AnsiString conversion in our framework. Faster, and compatible with Delphi 7 up to Delphi 2010 (didn't test with XE, but should pass).

I've already learnt not to depend on the Delphi RTL about string encoding. It's mandatory for having Open Source libraries working with multiple version of the Delphi compiler.

It's a pity that such regression occurs, in low-level part of the Delphi RTL.

Is there any other info about VCL / RTL enhancement, especially to prepare cross-platform? NativeUInt exists till Delphi 2007, but what about the Tag property of the TComponent (will it become NativeInt?), and so on...
0
Arnaud
9/1/2010 12:44:11 PM
Am 01.09.2010 14:44, schrieb Arnaud BOUCHEZ:

> (will it become NativeInt?)

I hope it's assured that the Tag property will be large enough to hold 
pointers on 64 bit machines...

Christian
0
Christian
9/1/2010 12:50:43 PM
Thanks for the information. I have raised QC 87709
0
Dave
9/1/2010 1:22:49 PM
Thanks for the suggestions

>   UnicodeToUtf8(szTitle, 511, PChar(WS), Cardinal(-1));

appears to effectively call the deprecated version of this function...

>   Result := UTF8Encode(WS).

This returns a RawByteString and I don't think this works with my database requirements for AnsiStrings (at least not without me worrying about any side effects)

I have gone with a FillChar(szTitle, 512, 0); before the conversion until the bug is fixed
0
Dave
9/1/2010 1:25:42 PM
Dave Speight wrote:

> Thanks for the suggestions
> 
> >   UnicodeToUtf8(szTitle, 511, PChar(WS), Cardinal(-1));
> 
> appears to effectively call the deprecated version of this function...

No it is the same function the -1 tells the used WideCharToMultiByte
function to add the terminating null character to the output string.
(and yes the deprecated function happens to do the same :) )
 
> I have gone with a FillChar(szTitle, 512, 0); before the conversion
> until the bug is fixed

That will work too :)

-- 
Pieter

"The police are not here to create disorder. They're here to
 preserve disorder."
 -- Ex-Chicago Mayor Daley during the 1968 riots
0
Pieter
9/1/2010 2:02:44 PM
<Dave Speight> wrote in message news:280985@forums.embarcadero.com...

> However, the UnicodeToUtf8 function appears to be badly
> broken making our app unusable.

The return value of UnicodeToUtf8() indicates the number of output 
characters.  In D2010, that includes a null terminator.  In XE, it does not. 
You should use the return value (even in D2010) instead of relying on the 
output buffer actually being null-terminated (the RTL's own code always 
looks the return value in both versions), eg:

{code:delphi}
function UTF8Str(const WS: string): AnsiString;
var
  szTitle: array[0..511] of AnsiChar;
  iLen: Cardinal;
begin
  szTitle[0] := #0;
  szTitle[3] := #1;
  iLen := UnicodeToUtf8(szTitle, 511, PChar(WS), Length(WS));
  if (iLen > 0) and (szTitle[iLen-1] = #0) then Dec(iLen);
  SetString(Result, szTitle, iLen);
end;
{code}

A better option is to use the overloaded version of Utf8Encode() that takes 
a UnicodeString as input and returns a RawByteString as output, and then 
copy that into your AnsiString.  Not only does that allow the RTL to encode 
the output correctly for that version's unique encoding semantics, but it 
also allows you to handle larger strings as well, eg:

{code:delphi}
function UTF8Str(const WS: string): AnsiString;
var
  szTitle: RawByteString;
begin
  szTitle := Utf8Encode(WS);
  SetString(Result, PAnsiChar(szTitle), Length(szTitle));
end;
{code}

A better option is to not encode the data manually at all (at least in D2009 
and later, anyway).  Assign the UnicodeString directly to a UTF8String 
variable and let the RTL decide how to encode it for you, eg:

{code:delphi}
function UTF8Str(const WS: string): AnsiString;
var
  szTitle: UTF8String;
begin
  szTitle := UTF8String(WS);
  SetString(Result, PAnsiChar(szTitle), Length(szTitle));
end;
{code}

--
Remy Lebeau (TeamB)
0
Remy
9/1/2010 7:10:21 PM
Thanks for the detailed and helpful information, one day I will hopefully get a full understanding of all the encoding options.

> {code:delphi}
>   szTitle := UTF8String(WS);
> {code}

I will go for this option as it seems the neatest. I remember trying all sorts of similar combinations of local variable types and encoding function calls to get code to put correctly encoded UTF8 in my database but I never thought to try the mysterious SetString function.

I assume you still agree there is an error in UnicodeToUtf8 as the doc promises it will provide a null-terminated sequence of UTF-8 characters. If so I will leave the QC entry.

Thanks,
Dave
0
Dave
9/2/2010 9:09:04 AM
<Dave Speight> wrote in message news:281598@forums.embarcadero.com...

>> {code:delphi}
>>   szTitle := UTF8String(WS);
>> {code}
>
> I will go for this option as it seems the neatest.

Just remember that UTF8String did not become true UTF-8 string until D2009. 
In earlier versions, UTF8String was just a typedef of AnsiString, so 
assigning a String to a UTF8String (or vice versa) was a reference-counted 
assignment and not an actual Ansi<->UTF8 conversation like it is in D2009+, 
so you will have to use UTF8Encode() (and UTF8Decode()) in pre-2009 
versions.

> I assume you still agree there is an error in UnicodeToUtf8 as the doc 
> promises
> it will provide a null-terminated sequence of UTF-8 characters. If so I 
> will leave
> the QC entry.

Since it is a good chunk of code that got removed, I assume it was 
intentional and they just forgot to update the comments.  This is why it is 
not a good idea to call system functions directly, as they are subject to 
change from version to version.  They are primarily meant for supporting the 
compiler/RTL, not the user.  Use higher-level coding (like using the 
UTF8String data type) and let the compiler/RTL decide how to use the 
lower-level system functions as needed.

-- 
Remy Lebeau (TeamB)
0
Remy
9/2/2010 5:32:38 PM
Reply:

Similar Artilces:

Delphi 7 to Delphi XE
Have been using Delphi 7 for many moons ( have got later versions but never upgraded to ) My first problem is: Component Palette. in XE it is a small toolbar docked in top right in Delphi 7 it gives a large view of all the components. I am struggling to be able to cope/access my components.in Delphi XE. Can I make the component pallette tool bar the same size as Delphi 7, or is there a fast way to view/choose all available components in XE, that I have not spotted yet? Kind Regards, Robert. Hi, What I know is that in Delphi 2010 and XE you can choose between t...

Delphi XE / Delphi 2010
Hello! I noticed that Embarcadero® Delphi® 2010 Version is not on the list of products on Embarcadero page. Or is it still possible to buy it? Will RAD Studio XE compile programs written in Delphi 2010 without problems.? Thanks. Am 13.09.2010 09:04, schrieb Petra Nemec: > Will RAD Studio XE compile programs written in Delphi 2010 without problems.? As always you will probably have to recreate the projects as the import is still a bit -- special. Christian Hello! Does anybody know if it is still possible to get a Delphi2010 trial version (if yes where)? ...

IBX (Delphi XE), Interbase XE and character set UTF8
Hello, I'm trying to get Interbase and IBX running with an UTF8 database. I'm using Delphi XE and Interbase XE. I'm creating the database like this: var aIBDatabase: TIBDatabase; ... aIBDatabase.DatabaseName := 'test_utf8.ib' aIBDatabase.Params.Clear; aIBDatabase.Params.Add('PAGE_SIZE 4096'); aIBDatabase.Params.Add('DEFAULT CHARACTER SET utf8'); aIBDatabase.CreateDatabase; The fields in the tables were created as VARCHAR. The database really seems to use character set utf8, because with IBExpert I'm able to set and r...

Migrating to Delphi XE from Delphi 7.0
Below is my code in Delphi 7.0, this is how to call another units in webmodule... Hello All, I create a web application in Delphi 7.0, using the Web Server Application, CGI, IntraWeb 7.0.15. And I used TIWPageProducer to view like this url "http://localhost/mcr/mcr.exe/main". I built and run. I viewed in thru IIS and it is running... This is my code in Delphi 7.0 .... .... procedure TWebModule1.proMainGetForm(ASender: TIWPageProducer; AWebApplication: TIWApplication; var VForm: TIWPageForm); begin VForm := TfrmMain.Create(AWebApplication); end; procedure TWebModule1....

Migrate from Delphi 2007 for Win32 to Delphi XE
we use Delphi 2007 for Win32 to support legacy (32Bit) OWL-based pascal applications (yes i know it was a mistake not to switch to VCL 15 years ago). could our applications still be opened and compiled with Delphi XE? The existing projects are all plain Pascal-Code, coming back from the times of Turbo Pascal for Windows and later on Borland Pascal. Are there any improvements we could profit from (i.e IDE, Debugger)? Thanks Andrej > {quote:title=Andrej Dimic wrote:}{quote} > could our applications still be opened and compiled with Delphi XE? I'm not sure, but I guess ...

Error on Delphi 6 but not on Delphi Xe for Ftp
I am Experimenting with get a file from our webside server via Ftp. I have 2 Machines 1 a laptop runing XP Delphi 6 Indy 10.5.8.0 An a machine runing Window 7 Delphi XE2 with Indy 10.5.8.0. I am using the Same Code on Both. procedure TFrmMain.ProcessItemDalySpecial; var PathDest : String; FileName : String; begin with FrmTb2 do begin if ReadIniBoolean(IniCfg,'FTP','UseFtpDaly') then begin Ftp.Host := ReadIniStr(IniCfg,'FTP','HostDaly'); Ftp.Port := ReadIniInt(IniCfg,'FTP'...

Migrating to Delphi XE from Delphi 7.0
Below is my code in Delphi 7.0, this is how to call another units in webmodule... Hello All, I create a web application in Delphi 7.0, using the Web Server Application, CGI, IntraWeb 7.0.15. And I used TIWPageProducer to view like this url "http://localhost/mcr/mcr.exe/main". I built and run. I viewed in thru IIS and it is running... This is my code in Delphi 7.0 .... .... procedure TWebModule1.proMainGetForm(ASender: TIWPageProducer; AWebApplication: TIWApplication; var VForm: TIWPageForm); begin VForm := TfrmMain.Create(AWebApplication); end;...

Migrating from Delphi 6 to Delphi XE 3! [Edit]
All, I am a Delphi developer working in an windows form application developed using Delphi 6. Now, we are planning to upgrade the development tool. Can anyone provide me information related to major roadblocks that we can face while migrating from Delphi 6 to Delphi XE 3? Should we migrate to Delphi XE 3 or any other preferred version of Delphi based on the fact that our target users will be using Windows 7 or Windows 8? Do we have any tools or utilities to migrate the source code from Delphi 6 to higher version of Delphi? Also, any suggestions related to best practices are welcome....

Delphi 7.0 code convert to delphi XE ...
Hello All, I create an application using Web Server Application then CGI stand alone... In WebModule I add ModuleController component and IWPageProcedure... Below is my code in Delphi 7.0, this is how to call another units in webmodule... .... .... procedure TWebModule1.proMainGetForm(ASender: TIWPageProducer; AWebApplication: TIWApplication; var VForm: TIWPageForm); begin VForm := TfrmMain.Create(AWebApplication); end; procedure TWebModule1.proLogInGetForm(ASender: TIWPageProducer; AWebApplication: TIWApplication; var VForm: TIWPageForm); begin VForm := TfrmLogIn.Create(AWebA...

Delphi 5 (string[80]) vs Delphi XE
Hi guys i'm trying to convert a old application made with Delphi 5 to Delphi XE 5 The probleme i have right now is about String[80] and the Read / Write procedure i have a type like that Type TEnrMess = record Mess : String[80]; MessSuiv : Word; Hint : Word; end; And i read/write this type to a File of TEnrMess My probleme is that the string have changed from Delphi 5 to Delphi XE i join the link of a simple of the probleme that can be compiled on delphi 5 and any Delphi XE https://mega.co.nz/#!wV1VyZzY!1ha5oOxTg7L1WJAOXutUygkj...

Migration from Delphi 2007 Rad to Delphi XE Where is Quickreports
Have installed and finally registered Delphi Xe, I am looking for Quickreports but am unable to find it anywhere, I asked the Distributer and was told its packaged in Delphi XE, Well I should have gone to specsavers......cos I cant see it any where ? > {quote:title=Colin Coleman wrote:}{quote} > Have installed and finally registered Delphi Xe, I am looking for Quickreports but am unable to find it anywhere, I asked the Distributer and was told its packaged in Delphi XE, Well I should have gone to specsavers......cos I cant see it any where ? Hello Colin, Delphi X...

Any difference between Delphi Prism 2011 and Delphi Prism XE?
Looking at the features in Delphi Prism XE, they look the same as the new items in the 2011 release back in may. I there anything new in the XE release? or did they simply change the product branding? Just wondering if I need to update it or now when i download the rest. Thanks, Hi Dan, > Looking at the features in Delphi Prism XE, they look the same as the new items in the 2011 release back in may. I there anything new in the XE release? or did they simply change the product branding? Just wondering if I need to update it or now when i download the rest. See http://w...

Delphi 7 to Delphi XE: TBlobField to XML [Edit]
Hi, I'm migrating a Delphi7 application to Delphi XE. I'm using a TClientDataSet to communicate, by using a XML frame, with my server. In this TClientDataSet I'm using a TBlobField which is an array of 384 byte. The blobField is allocate by a code like this : {code} myStream : TStream; myStream := aClientDataSet.CreateBlobStream(myBlobField, bmwrite); vResult := myStream.Write(ArrayOf384Byte[0], length(ArrayOf384Byte)); //vResult = 384 => GooD ! (...) {code} For communicate with the server, we have to decode the Blobfield in XML before to sending it. We have...

Delphi and Delphi for .Net
It seems that Delphi for .Net is slower than Delphi Win32 native applicaiton. I would like to know is it true all .Net application is slower than Win32 native applicaiton or it is Delphi for .Net only. Your information is great appreciated, Inung On 2011-06-21 18:20:17 +0100, Inung Huang said: > It seems that Delphi for .Net is slower than Delphi Win32 native applicaiton. > I would like to know is it true all .Net application is slower than > Win32 native applicaiton or it is Delphi for .Net only. If you are only running the code in the application once then, yes, yo...

Web resources about - Delphi XE broken UTF8 - embarcadero.delphi.general

Delphi - Wikipedia, the free encyclopedia
... an archaeological site and a modern town in Greece on the south-western spur of Mount Parnassus in the valley of Phocis . The site of Delphi ...

Delphi Automotive (@DelphiAuto) on Twitter
Log in Sign up You are on Twitter Mobile because you are using an old version of Internet Explorer. Learn more here Delphi Automotive @ DelphiAuto ...

Delphi Connect for Verizon on the App Store on iTunes
Get Delphi Connect for Verizon on the App Store. See screenshots and ratings, and read customer reviews.


Audi working with Delphi to develop autonomous car tech
Audi is developing an iPad-sized device that will pack all the necessary computing power for a self-driving car

US approves China company's acquisition of Delphi biz
The Committee on Foreign Investment in the United States has formally approved the acquisition of Delphi's global production of braking systems ...

Verizon And Delphi Officially Launch Vehicle Diagnostics Service - $250 For The Module, $5 A Month On ...
If you're a car nut, a paranoid parent, or a small business owner looking to do a little, uh, company vehicle economy analysis, Verizon's teamed ...

Watch out Google: Delphi gives Ars a ride in its self-driving car
The automotive components maker gave Ars a preview ride around the neighborhood. MOUNTAIN VIEW, CA—On Thursday morning I met with Delphi at its ...

The skinny on Delphi's autonomous road trip across the United States
Filed under: Green , Videos , Autonomous Last week, Delphi's autonomous car became the first to complete a coast-to-coast trip across the United ...

Delphi partners with WiTricity on automated wireless charging system
One could easily argue that parking between the white lines at any local hangout presents a challenge for some inexperienced drivers. So, why ...

Resources last updated: 1/22/2016 4:52:53 AM