(Net::LDAP) Automatically convert attributes into utf8 when writting

Hi, we are using an old version of Net::LDAP (0.39) in an old perl
installation (5.10.1). Recently we have changed the ldap server, and now it 
uses utf8 in the entry attributes, so we are getting problems with reading and 
writting attributes with Net::LDAP.

To solve it, I have read the documentation of Net::LDAP 0.39 ( at
https://metacpan.org/pod/release/GBARR/perl-ldap-0.39/lib/Net/LDAP.pod ), and
I see there is an option ("raw") in the constructor to indicate attributes
that should be treated as utf8. I have tested it, and it works por reading
from the ldap server (attribute strings are marked as utf8, so they a treated
correctly by our programs), but it doesn't work for writting (our latin1 
strings are not being converted automatically into utf8 before being sent). 

So it looks like the "raw" option works for reading but not for writting. Is
there any quick way to use the "raw" regex also for writting? The alternative
would be to review all of our code and manually encode all the values to utf8
before passing them to Net::LDAP, but it would mean a lot of work. It would be
better if we could change the Net::LDAP library itself to convert
automatically attributes into utf8, same as for reading. For example, a new 
option "raw_for_writing" could be added to the constructor:

������$ldap = Net::LDAP->new(
��������������������������������$server, 
��������������������������������port => $port,
��������������������������������raw => qr/(?i:^jpegPhoto|;binary)/,
��������������������������������raw_for_writing => 1,
����������������������������)

I see that the automatic conversion for reading is done at the "decode"
function of "Entry.pm":

����sub decode {

������...

������if (CHECK_UTF8 && $arg{raw}) {
��������$result->{objectName} = Encode::decode_utf8($result->{objectName})
����������if ('dn' !~ /$arg{raw}/);

������...

��������foreach my $elem (@{$self->{asn}{attributes}}) {
����������map { $_ = Encode::decode_utf8($_) } @{$elem->{vals}}
������������if ($elem->{type} !~ /$arg{raw}/);
��������}
������}

And I see that there is an "encode" function in "Entry.pm", that doesn't do
the magic:

����sub encode {
������$LDAPEntry->encode( shift->{asn} );
����}

Would it be sufficient to add some similar code to the "Entry::encode" 
function in order to automatically encode attributes to utf8 before being sent?

Any suggestion to reduce the amount of code to be changed in our programs?

Thank you
0
pe
8/25/2015 10:54:26 AM
perl.ldap 1268 articles. 0 followers. Follow

4 Replies
633 Views

Similar Articles

[PageSpeed] 11

Hello,
  instead of patching Net::LDAP you should use utf8::encode() and
utf8::decode() in your perl code.

See http://perldoc.perl.org/5.10.1/utf8.html .

Regards,  Jochen.

Am 25.08.2015 um 12:54 schrieb pe rl:
> Hi, we are using an old version of Net::LDAP (0.39) in an old perl
> installation (5.10.1). Recently we have changed the ldap server, and now it 
> uses utf8 in the entry attributes, so we are getting problems with reading and 
> writting attributes with Net::LDAP.
>
> To solve it, I have read the documentation of Net::LDAP 0.39 ( at
> https://metacpan.org/pod/release/GBARR/perl-ldap-0.39/lib/Net/LDAP.pod ), and
> I see there is an option ("raw") in the constructor to indicate attributes
> that should be treated as utf8. I have tested it, and it works por reading
> from the ldap server (attribute strings are marked as utf8, so they a treated
> correctly by our programs), but it doesn't work for writting (our latin1 
> strings are not being converted automatically into utf8 before being sent). 
>
> So it looks like the "raw" option works for reading but not for writting. Is
> there any quick way to use the "raw" regex also for writting? The alternative
> would be to review all of our code and manually encode all the values to utf8
> before passing them to Net::LDAP, but it would mean a lot of work. It would be
> better if we could change the Net::LDAP library itself to convert
> automatically attributes into utf8, same as for reading. For example, a new 
> option "raw_for_writing" could be added to the constructor:
>
>       $ldap = Net::LDAP->new(
>                                 $server, 
>                                 port => $port,
>                                 raw => qr/(?i:^jpegPhoto|;binary)/,
>                                 raw_for_writing => 1,
>                             )
>
> I see that the automatic conversion for reading is done at the "decode"
> function of "Entry.pm":
>
>     sub decode {
>
>       ...
>
>       if (CHECK_UTF8 && $arg{raw}) {
>         $result->{objectName} = Encode::decode_utf8($result->{objectName})
>           if ('dn' !~ /$arg{raw}/);
>
>       ...
>
>         foreach my $elem (@{$self->{asn}{attributes}}) {
>           map { $_ = Encode::decode_utf8($_) } @{$elem->{vals}}
>             if ($elem->{type} !~ /$arg{raw}/);
>         }
>       }
>
> And I see that there is an "encode" function in "Entry.pm", that doesn't do
> the magic:
>
>     sub encode {
>       $LDAPEntry->encode( shift->{asn} );
>     }
>
> Would it be sufficient to add some similar code to the "Entry::encode" 
> function in order to automatically encode attributes to utf8 before being sent?
>
> Any suggestion to reduce the amount of code to be changed in our programs?
>
> Thank you

0
mlists
8/25/2015 11:04:21 AM
Thank you, I already knew utf8::encode() and utf8::decode().

They are not necessary when reading/searching in the ldap server, since 
Net::LDAP already has a "raw" option in the constructor to automatically
encode/decode strings. It is working for us, and the only change required has 
been to add the "raw" option to the constructor.

The problem appears when writting to the ldap server. I have started to modify
our code with utf8::encode(), by adding it to every attribute in all of our
functions. The problem is that it is very inefficient, since I will have to 
modify every attribute that appears in our programs. We have a lot of functions 
that create/modify/delete entries in the ldap server, so I will have to change 
a lot of code to manually encode attribs to utf8, and then test all of the 
changes.

It would be much simpler if Net::LDAP would encode automatically the
attributes by using the regex passed into the "raw" option of the constructor,
since the changes in our programs would be zero. In my first message I pasted
the code in Net::LDAP that encodes the attributes when reading from the ldap
server, and it looks simple. Probably encoding attributes when writting to the
ldap server could be simple as well. Probably the changes required in
Net::LDAP are minimal compared to the changes required in our code.

Thank you


25.08.2015, 13:04, "Keutel, Jochen (mlists)" <mlists@keutel.de>:
> Hello,
> ��instead of patching Net::LDAP you should use utf8::encode() and
> utf8::decode() in your perl code.
>
> See http://perldoc.perl.org/5.10.1/utf8.html .
>
> Regards, Jochen.
>
> Am 25.08.2015 um 12:54 schrieb pe rl:
>> �Hi, we are using an old version of Net::LDAP (0.39) in an old perl
>> �installation (5.10.1). Recently we have changed the ldap server, and now it
>> �uses utf8 in the entry attributes, so we are getting problems with reading and
>> �writting attributes with Net::LDAP.
>>
>> �To solve it, I have read the documentation of Net::LDAP 0.39 ( at
>> �https://metacpan.org/pod/release/GBARR/perl-ldap-0.39/lib/Net/LDAP.pod ), and
>> �I see there is an option ("raw") in the constructor to indicate attributes
>> �that should be treated as utf8. I have tested it, and it works por reading
>> �from the ldap server (attribute strings are marked as utf8, so they a treated
>> �correctly by our programs), but it doesn't work for writting (our latin1
>> �strings are not being converted automatically into utf8 before being sent).
>>
>> �So it looks like the "raw" option works for reading but not for writting. Is
>> �there any quick way to use the "raw" regex also for writting? The alternative
>> �would be to review all of our code and manually encode all the values to utf8
>> �before passing them to Net::LDAP, but it would mean a lot of work. It would be
>> �better if we could change the Net::LDAP library itself to convert
>> �automatically attributes into utf8, same as for reading. For example, a new
>> �option "raw_for_writing" could be added to the constructor:
>>
>> �������$ldap = Net::LDAP->new(
>> ���������������������������������$server,
>> ���������������������������������port => $port,
>> ���������������������������������raw => qr/(?i:^jpegPhoto|;binary)/,
>> ���������������������������������raw_for_writing => 1,
>> �����������������������������)
>>
>> �I see that the automatic conversion for reading is done at the "decode"
>> �function of "Entry.pm":
>>
>> �����sub decode {
>>
>> �������...
>>
>> �������if (CHECK_UTF8 && $arg{raw}) {
>> ���������$result->{objectName} = Encode::decode_utf8($result->{objectName})
>> �����������if ('dn' !~ /$arg{raw}/);
>>
>> �������...
>>
>> ���������foreach my $elem (@{$self->{asn}{attributes}}) {
>> �����������map { $_ = Encode::decode_utf8($_) } @{$elem->{vals}}
>> �������������if ($elem->{type} !~ /$arg{raw}/);
>> ���������}
>> �������}
>>
>> �And I see that there is an "encode" function in "Entry.pm", that doesn't do
>> �the magic:
>>
>> �����sub encode {
>> �������$LDAPEntry->encode( shift->{asn} );
>> �����}
>>
>> �Would it be sufficient to add some similar code to the "Entry::encode"
>> �function in order to automatically encode attributes to utf8 before being sent?
>>
>> �Any suggestion to reduce the amount of code to be changed in our programs?
>>
>> �Thank you
0
pe
8/25/2015 11:37:15 AM
Hi,

On Tuesday, 25. August 2015 13:37:15 pe rl wrote:
> They are not necessary when reading/searching in the ldap server, since
> Net::LDAP already has a "raw" option in the constructor to automatically
> encode/decode strings. It is working for us, and the only change required
> has been to add the "raw" option to the constructor.

I think you misinterpret the purpose of the raw option.

Its goal is to convert the byte strings coming from the LDAP server that 
represent UTF-8 encoded directory strings from byte semantics to
Perl scalars with character semantics.

On the other hand, perl-ldap expects scalars in character semantics when
it comes to writing directory strings to an LDAP server.

It is not perl-ldap's job to translate between scalars in Perl's character 
semantics and various input or output encodings of your application.


> The problem appears when writting to the ldap server. I have started to
> modify our code with utf8::encode(), by adding it to every attribute in all
> of our functions. The problem is that it is very inefficient, since I will
> have to modify every attribute that appears in our programs. We have a lot
> of functions that create/modify/delete entries in the ldap server, so I
> will have to change a lot of code to manually encode attribs to utf8, and
> then test all of the changes.

It is not perl-ldap's job to translate between scalars in Perl's character 
semantics and various input or output encodings of your application.

This is the application's task.
If you - as you write - need to convert every attribute using ut8::encode(),
then your application seems to use a mixture of byte & character semantics.

In that case please do yourself a favour and switch over to character 
semantics by correctly converting input to character semantics when it 
happens:
- for file & console input you can use the ":encoding(...)" layer to make
  sure you get character semantics instead of byte semantics
- for @ARGV a simple
     $_ = Encode::decode('UTF-8' ,$_) for @ARGV;
  should be sufficient.

You may also have a look at the 'utf8::all' package that does a lot of the 
above for you automatically.

Please read the perlunicode manual page for more detailed information.

Best
PEter

-- 
Peter Marschall
peter@adpm.de

0
peter
8/29/2015 11:54:15 AM
Thank you for your information.

Finally I added "uf8::encode" to all the attribs, so now it works.

Converting our code (@_ and file i/o) into utf8 was an option, but I discarded it because we have a lot of files (our proyect is nearly a framework, not a few files), including modules that read translation string files for several languages, so converting everything into utf8 would be a lot of extra work.

Our proyect is rather old, it was created in the old times, when utf8 was still not used. This is the reason why it is so difficult for us to convert everyting into utf8. Anyway I believe we will have to convert it some day, as you proposed.

Thank you


29.08.2015, 13:54, "Peter Marschall" <peter@adpm.de>:
> Hi,
>
> On Tuesday, 25. August 2015 13:37:15 pe rl wrote:
>> �They are not necessary when reading/searching in the ldap server, since
>> �Net::LDAP already has a "raw" option in the constructor to automatically
>> �encode/decode strings. It is working for us, and the only change required
>> �has been to add the "raw" option to the constructor.
>
> I think you misinterpret the purpose of the raw option.
>
> Its goal is to convert the byte strings coming from the LDAP server that
> represent UTF-8 encoded directory strings from byte semantics to
> Perl scalars with character semantics.
>
> On the other hand, perl-ldap expects scalars in character semantics when
> it comes to writing directory strings to an LDAP server.
>
> It is not perl-ldap's job to translate between scalars in Perl's character
> semantics and various input or output encodings of your application.
>
>> �The problem appears when writting to the ldap server. I have started to
>> �modify our code with utf8::encode(), by adding it to every attribute in all
>> �of our functions. The problem is that it is very inefficient, since I will
>> �have to modify every attribute that appears in our programs. We have a lot
>> �of functions that create/modify/delete entries in the ldap server, so I
>> �will have to change a lot of code to manually encode attribs to utf8, and
>> �then test all of the changes.
>
> It is not perl-ldap's job to translate between scalars in Perl's character
> semantics and various input or output encodings of your application.
>
> This is the application's task.
> If you - as you write - need to convert every attribute using ut8::encode(),
> then your application seems to use a mixture of byte & character semantics.
>
> In that case please do yourself a favour and switch over to character
> semantics by correctly converting input to character semantics when it
> happens:
> - for file & console input you can use the ":encoding(...)" layer to make
> ��sure you get character semantics instead of byte semantics
> - for @ARGV a simple
> �����$_ = Encode::decode('UTF-8' ,$_) for @ARGV;
> ��should be sufficient.
>
> You may also have a look at the 'utf8::all' package that does a lot of the
> above for you automatically.
>
> Please read the perlunicode manual page for more detailed information.
>
> Best
> PEter
>
> --
> Peter Marschall
> peter@adpm.de
0
pe
8/31/2015 7:42:09 AM
Reply:

Similar Artilces:

(Net::LDAP) Automatically convert attributes into utf8 when writting #2
Hi, we are using an old version of Net::LDAP (0.39) in an old perl installation (5.10.1). Recently we have changed the ldap server, and now it uses utf8 in the entry attributes, so we are getting problems with reading and writting attributes with Net::LDAP. To solve it, I have read the documentation of Net::LDAP 0.39 ( at https://metacpan.org/pod/release/GBARR/perl-ldap-0.39/lib/Net/LDAP.pod ), and I see there is an option ("raw") in the constructor to indicate attributes that should be treated as utf8. I have tested it, and it works por reading from the ldap server (attr...

How do I retrieve operational attributes for an LDAP entry using Net::LDAP?
how do I retrieve the values for 'creatorsName','createTimestamp', 'modifiersName', 'modifyTimestamp' using Net::LDAP module? pleae help ===== use Net::LDAP; use Net::LDAP::Util qw(ldap_error_text ldap_error_name ldap_error_desc); $host='xxxx.com'; $rdn='cn=manager,dc=xxxx,dc=com'; $ldappasswd='123456'; my $ldap=new Net::LDAP($host) or die; my $mesg=$ldap->bind("$rdn",password=>"$ldappasswd",version => 3) or die; my $mesg=$ldap->search(base=>"ou=people,dc=xxxx,dc=com",scope=>...

$ldap->search triggers Bug in ASN1 and utf8 Net::LDAP code
Dear perl programmers, I think I found bug in Net::LDAP in the part which is doing something with ASN.1 (Convert::ASN1). I'm writing small program for selecting and exporting part of oracle (with DBD::Oracle) doing nasty things with output, selecting from OpenLDAP server matching and non-matching entries and comparing them attribute by attribute, making ldif diff file from that, or producing new entries if $mesg->entries is 0. I'm not a professional programmer, but I *must* finish this application. :-( Output from oracle is in UTF-8, and it has many names with nati...

make Net::LDAP::LDIF more similar to Net::LDAP
Hi Graham, hi Chris, hi list, I would like to rework Net::LDAP::LDIF a bit so that its API resembles that of Net::LDAP a bit more while still keeping the traditional API. The reason for this is that in application I often need to distinguish between Net::LDAP and Net::LDAP::LDIF because some methods are only implemented on one side. I\'d like to start with a code() method that tries to mimic the Net::LDAP one and I\'d like to extend the Net::LDAP::Entry->update() method so that it takes a Net::LDAP::LDIF object as an argument. The latter one requires a...

LDAP attribute Map / LIst / extend the LDAP attributes
we are use ladp on netware 65, is there a list of the LDAP attributes avaliable that are used for eDirectory 8.7? is it possible to create a ldap attribute that contains more that one edirectory attribute content and extend it with a static variable? any ideas HELGE -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Most eDirectory attributes are available natively by their name (minus spaces and special characters). For example fullname works to retrieve the 'Full Name' and givenname works for 'Given Name' and sasloginconfiguration works for 'SAS:...

[Fwd: make Net::LDAP::LDIF more similar to Net::LDAP]
--------------95D5815B06BDC2BD1A0ABFEB Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit --------------95D5815B06BDC2BD1A0ABFEB Content-Type: message/rfc822 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Mozilla-Status2: 00000000 Message-ID: <40C7B13E.8864E5A0@cs.adelaide.edu.au> Date: Thu, 10 Jun 2004 10:54:22 +1000 From: Sion Camilleri <sion@cs.adelaide.edu.au> Reply-To: sion@cs.adelaide.edu.au X-Mailer: Mozilla 4.8 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Graham Barr <gbarr@pobox.com>...

Net::LDAP v0.28, bug in Net::LDAP::Constant, :all not supported
Net::LDAP::Constant no longer supports the :all tag in the export list due to the switch from Exporter to a manual export routine. So, while the following: perl -MNet::LDAP::Constant=:all -e 1 worked fine in 0.2701, it now dies with the error: ":all" is not exported by the Net::LDAP::Constant module at -e line 0 Can't continue after import errors at -e line 0 BEGIN failed--compilation aborted, <DATA> line 197. The documentation for Net::LDAP::Constant still documents the ':all' tag. I am not subscribed to the list, so if some...

Net::LDAP and Net:LDAP::LDIF read & add problems
I'm trying to read in a simple LDIF file to add an entry to my LDAP server. Here is the basic routine (extraneous details omitted for brevity and security): $ldif = Net::LDAP::LDIF->new($tmp,"r",onerror => 'warn'); $entry = $ldif->ready_entry(); $ldap = Net::LDAP->new($LDAPSERVER); $result=$ldap->bind("$binddn",password=>"$bindpass",version=>"3"); $result=$ldap->add($entry); Now, everything seems to work until I get to the $ldap->add method. From that I get various versions of the following: object...

Fw: Re: make Net::LDAP::LDIF more similar to Net::LDAP
Am 08.06.2004 um 18:29 Uhr haben Sie geschrieben: > On 8 Jun 2004, at 16:56, peter@adpm.de wrote: > > I\'d like to start with a code() method that tries to mimic the >> Net::LDAP one and >I assume you mean better error handling ? My first goal is having a code() method in Net::LDAP::LDIF. > > I\'d like to extend the > > Net::LDAP::Entry->update() method so that it takes a > > Net::LDAP::LDIF object as an argument. The latter one > > requires a bit of work in Net::LDAP::LDIF to make it > > correct. > Not su...

Fw: Re: make Net::LDAP::LDIF more similar to Net::LDAP #2
Hi, Am 09.06.2004 um 01:44 Uhr haben Sie geschrieben: > Extending Net::LDAP::Entry to update against LDIF and LDAP objects > could allow the changetype modifications to be to produced. > > This would be really useful to produce changetypes for entry objects by > updating against an LDIF object to produce the changetype LDIF required > up to synchronise entry objects. as Graham posted you can to that already now. Simply create your Net::LDAP::Entry object with the changes option set to TRUE. Having created the ::LDIF object that way you autom...

Net::LDAP and Net:LDAP::LDIF read & add problems #2
I'm trying to read in a simple LDIF file to add an entry to my LDAP server. Here is the basic routine (extraneous details omitted for brevity and security): $ldif = Net::LDAP::LDIF->new($tmp,"r",onerror => 'warn'); $entry = $ldif->ready_entry(); $ldap = Net::LDAP->new($LDAPSERVER); $result=$ldap->bind("$binddn",password=>"$bindpass",version=>"3"); $result=$ldap->add($entry); Now, everything seems to work until I get to the $ldap->add method. From that I get various versions of the following: obj...

One or more eDir to LDAP attribute mappings appear to be incorrect. Change attribute mappings through the LDAP
--____IYZLTTEASTICDGWKXZWG____ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Content-Disposition: inline; modification-date="Fri, 6 Nov 2008 16:52:05 +0100" SGkuDQoNClJ1bm5pbmcgWkVOd29ya3MgTWlncmF0aW9uIFV0aWxpdHkgdjEwLjEuMS4wIGFuZCB0 cnlpbmcgdG8gTWlncmF0ZSBBcHBsaWNhdGlvbnMuDQoNCkJ1dCBqdXN0IGdldHMgdGhpcyBFcnJv ci4uLg0KT25lIG9yIG1vcmUgZURpciB0byBMREFQIGF0dHJpYnV0ZSBtYXBwaW5ncyBhcHBlYXIg dG8gYmUgaW5jb3JyZWN0LiBDaGFuZ2UgYXR0cmlidXRlIG1hcHBpbmdzIHRocm91Z2ggdGhlIExE QVAgDQoNCldpdGNoIHNob3VsZCBiZSBmaXhlZCBpbiB2ZXJzaW9uIDEwLjAuMyByZWdhcmRpbmcg dG8gV...

Net:Net:Net::LDAP::FAQ
------_=_NextPart_001_01C6429F.D89AA417 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, Net::LDAP Net::LDAPS Is there a possible to LDAP bind with an encrypted (SHA, SSHA, CRYPT, ....) password? I don't like to write the secret password to the perl file. Best regards Barbara Wilbert ------_=_NextPart_001_01C6429F.D89AA417-- Wilbert Barbara (CI/OSI) * wrote: > Hello, > > Net::LDAP > Net::LDAPS > > Is there a possible to LDAP bind with an encrypted (SHA, SSHA, CRYPT, > ......

LDAP export
I am trying to do an ldap export of certain user attribute fields, one of them being department. I can not find the ldap attribute mapping for the nds attribute called department. Is there a more extensive list out there, other than the one you see in ConsoleOne? Or does anyone know how the nds attribute: department maps to an ldap attribute? Thanks lhines@flir.com, The NDS attribute for Department is 'OU' and that should map to the LDAP attribute 'ou'. If you need more help with LDAP, please post to the novell.support.ds.ldap forum. -- //N...

Web resources about - (Net::LDAP) Automatically convert attributes into utf8 when writting - perl.ldap

Facebook Users Automatically Checked In To Events They RSVPed Yes To
A reader tipped us off that Facebook is automatically checking in users at events that they RSVPed they would attend. continued… New Career ...

Now Users Can Remove Contacts Automatically Saved by Facebook’s Friend Finder
Two weeks ago, many Facebook users began asking questions about curiously good recommendations suddenly appearing in Facebook’s “People You May ...

App Store - Attachments.me- Gmail inbox software to efficiently manage emails, automatically send/upload ...
Get Attachments.me- Gmail inbox software to efficiently manage emails, automatically send/upload files to cloud storage(Dropbox, Box, and G Drive), ...

Automatically organize your desktop icons into shaded areas called Fences! - YouTube
Fences® is the most popular desktop organization tool used by millions of users worldwide. Create shaded areas called "fences" to automatically ...

CSAIL fixes software bugs automatically, in any language, by copying from safer applications
A new system can repair bugs in software using smart processing that imports functionality from other programs, all without access to source ...


Emailing porn at work not automatically sackable, court finds
Australia's federal court has upheld a ruling that emailing pornography in the workplace is not automatically a sackable offence.

Shazam iPhone app now listens for music, TV shows automatically
Shazam has updated its iPhone app to tag songs, TV shows and more on its own, no longer requiring users to open the app and tap a button.

App of the day: Human for iPhone automatically tracks your movements
Human for the iPhone is an activity tracker that automatically distinguishes between different types of movement.

iPhone 5 automatically rotates using Cycloramic App. - YouTube
[NEW VIDEO] Cycloramic 2.0 update teaser vid with panoramic photo preview: http://www.youtube.com/watch?v=cjHUID07xs4 Cycloramic has been Awarded ...

Resources last updated: 12/18/2015 7:33:04 AM