updating POSIX module for newer functions

Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
but I figure it'll be useful to support new functionality...

POSIX::spawn would be good for starting background processes w/o
pipes (or multiple pipes), even if our popen may use posix_spawn(3)
transparently: http://nntp.perl.org/group/perl.perl5.porters/256771

Other functions I'd like to see in the POSIX module:

* pwrite/pread - useful for sharing handles across multiple processes
  or threads (implementing DBs and such).  pread is also useful for
  single-threaded file serving the same file thousands of clients,
  especially with sendfile being unusable with userspace TLS.

* posix_fadvise - gives the ability to manipulate kernel page cache usage,
  great for reducing cache pressure on local filesystem search indexers
  and also to prevent HDD spinups on laptops.

For comparison, all of the above are implemented in recent-ish
versions of Ruby.  Ruby's Process.spawn is a superset of
posix_spawn(3), though, and I'd like work on getting the POSIX
version to feature parity at some point...


A few more newer POSIX.1-2008 ones (not in Ruby):

* _AT_FILE functions: openat, renameat, unlinkat, mkdirat, etc.
  These are useful for:

  - preventing FS mounts from changing while a handle is open
  - reducing name lookup times in the kernel for deeply nested dirs
  - implementing per-thread working directories

* posix_fallocate - can be used to reduce fragmentation when
  writing big files

* statvfs/fstatvfs - yes Filesys::Statvfs exists, but I prefer
  to have to document less dependencies and to have fewer DSOs:
  https://udrepper.livejournal.com/8790.html

....

I'd be happy to implement any of these and post patches and pull
requests to a self-hosted git server.  I don't agree to GitHub's
terms-of-service nor will I run JavaScript to get past CAPTCHAs.

Thanks for reading.
0
p5p
12/16/2019 7:42:57 PM
perl.perl5.porters 48111 articles. 1 followers. Follow

8 Replies
82 Views

Similar Articles

[PageSpeed] 5

* Eric Wong (p5p@yhbt.net) [191216 19:47]:
> Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
> but I figure it'll be useful to support new functionality...

There are about 1800 functions in the 2008 release.  POSIX.pm
and CORE together support about 100 (but also not really strict) 

There are some modules on CPAN which implement some additional
functions.  A few dozen.

I started POSIX::1003 which tries to provide all functions... as
pure as feasible.  But... I need to rework it into a more flexible
installation structure using Config::AutoConf.  Help welcome ;-)
When I get help, I will increase its priority.
-- 
Regards,

               MarkOv

------------------------------------------------------------------------
       Mark Overmeer MSc                                MARKOV Solutions
       Mark@Overmeer.net                          solutions@overmeer.net
http://Mark.Overmeer.net                   http://solutions.overmeer.net
0
mark
12/16/2019 8:41:31 PM
--0000000000001a4f890599d8b05e
Content-Type: text/plain; charset="UTF-8"

On Mon, Dec 16, 2019 at 2:51 PM Eric Wong <p5p@yhbt.net> wrote:

> Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
> but I figure it'll be useful to support new functionality...
>

If it's at all helpful, it's trivial to write a FFI::Platypus wrapper of
POSIX functions available on the system, such as I did in Unix::Groups::FFI.

-Dan

--0000000000001a4f890599d8b05e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Mon, Dec 16, 2019 at 2:51 PM Eric Wong=
 &lt;<a href=3D"mailto:p5p@yhbt.net">p5p@yhbt.net</a>&gt; wrote:<br></div><=
div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin=
:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"=
>Hello all, I noticed POSIX.pod states it&#39;s for IEEE Std 1003.1,<br>
but I figure it&#39;ll be useful to support new functionality...<br></block=
quote><div><br></div><div>If it&#39;s at all helpful, it&#39;s trivial to w=
rite a FFI::Platypus wrapper of POSIX functions available on the system, su=
ch as I did in Unix::Groups::FFI.</div><div><br></div><div>-Dan</div></div>=
</div>

--0000000000001a4f890599d8b05e--
0
grinnz
12/16/2019 9:13:38 PM
> On Mon, Dec 16, 2019 at 2:51 PM Eric Wong <p5p@yhbt.net> wrote:
> 
> > Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
> > but I figure it'll be useful to support new functionality...

Hi Mark, Dan; thanks for the responses so far.

Some background here: most users of my software get Perl from
their GNU/Linux distros or BSD ports.  And a big reason why I
choose Perl is that it's widely available out-of-the-box and
users typically don't need to install extra dependencies.

CPAN modules which aren't packaged by the distro are extra
overhead for them to learn, download and install; and they
already complain about dependencies.  Most of them are not Perl
hackers.  Even getting them to consider using software written
in Perl these days is a challenge :<

Mark Overmeer <mark@overmeer.net> wrote:
> There are about 1800 functions in the 2008 release.  POSIX.pm
> and CORE together support about 100 (but also not really strict) 
> 
> There are some modules on CPAN which implement some additional
> functions.  A few dozen.

Right, some are abandoned and most of those are not available
from distros.

> I started POSIX::1003 which tries to provide all functions... as
> pure as feasible.  But... I need to rework it into a more flexible
> installation structure using Config::AutoConf.  Help welcome ;-)
> When I get help, I will increase its priority.

Any reason you choose to work on a new module rather than
improving the existing POSIX one?  I certainly don't see the
need for all functions in POSIX, especially the ones which
are made redundant by CORE functions.

Dan Book <grinnz@gmail.com> wrote:
> If it's at all helpful, it's trivial to write a FFI::Platypus wrapper of
> POSIX functions available on the system, such as I did in Unix::Groups::FFI.

Yup.  I favor Inline(::C) since it's more widely-packaged
(for example, perl-Inline is in CentOS/RHEL 7.x and AFAIK
Platypus is not).

I may even favor syscall() for popular platforms since there's
no extra .so loading and no need to have a compiler installed.

But I consider those dependencies interim solutions because of
the extra installation burden it places on users (and distro
maintainers).  And this probably a big reason why AOT-compiled
languages like Go with giant binaries are gaining favor
nowadays.
0
p5p
12/17/2019 7:18:38 AM
* Eric Wong (p5p@yhbt.net) [191217 07:18]:
> > I started POSIX::1003 which tries to provide all functions... as
> > pure as feasible.  But... I need to rework it into a more flexible
> > installation structure using Config::AutoConf.  Help welcome ;-)
> > When I get help, I will increase its priority.
> 
> Any reason you choose to work on a new module rather than
> improving the existing POSIX one?  I certainly don't see the
> need for all functions in POSIX, especially the ones which
> are made redundant by CORE functions.

Everyone has different needs, so the easiest way to make the such a
module useful is by providing everything without exception.  All in
one distro, which is easier to maintain for the packagers.

It is often unclear which smart behavior a CORE function has.  For
instance, which function is compiled in for the setuid().  When you
are really into high quality OS programming, you have to know.
-- 
greetz,

               MarkOv

------------------------------------------------------------------------
       Mark Overmeer MSc                                MARKOV Solutions
       Mark@Overmeer.net                          solutions@overmeer.net
http://Mark.Overmeer.net                   http://solutions.overmeer.net
0
mark
12/17/2019 7:32:03 AM
On 12/17/19 9:18 AM, Eric Wong wrote:
>> On Mon, Dec 16, 2019 at 2:51 PM Eric Wong <p5p@yhbt.net> wrote:
>>
>>> Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
>>> but I figure it'll be useful to support new functionality...
> Hi Mark, Dan; thanks for the responses so far.
>
> Some background here: most users of my software get Perl from
> their GNU/Linux distros or BSD ports.  And a big reason why I
> choose Perl is that it's widely available out-of-the-box and
> users typically don't need to install extra dependencies.
>
> CPAN modules which aren't packaged by the distro are extra
> overhead for them to learn, download and install; and they
> already complain about dependencies.  Most of them are not Perl
> hackers.  Even getting them to consider using software written
> in Perl these days is a challenge :<
>
> Mark Overmeer <mark@overmeer.net> wrote:
>> There are about 1800 functions in the 2008 release.  POSIX.pm
>> and CORE together support about 100 (but also not really strict) 
>>
>> There are some modules on CPAN which implement some additional
>> functions.  A few dozen.
> Right, some are abandoned and most of those are not available
> from distros.
>
>> I started POSIX::1003 which tries to provide all functions... as
>> pure as feasible.  But... I need to rework it into a more flexible
>> installation structure using Config::AutoConf.  Help welcome ;-)
>> When I get help, I will increase its priority.
> Any reason you choose to work on a new module rather than
> improving the existing POSIX one?  I certainly don't see the
> need for all functions in POSIX, especially the ones which
> are made redundant by CORE functions.
>
> Dan Book <grinnz@gmail.com> wrote:
>> If it's at all helpful, it's trivial to write a FFI::Platypus wrapper of
>> POSIX functions available on the system, such as I did in Unix::Groups::FFI.
> Yup.  I favor Inline(::C) since it's more widely-packaged
> (for example, perl-Inline is in CentOS/RHEL 7.x and AFAIK
> Platypus is not).
>
> I may even favor syscall() for popular platforms since there's
> no extra .so loading and no need to have a compiler installed.
>
> But I consider those dependencies interim solutions because of
> the extra installation burden it places on users (and distro
> maintainers).  And this probably a big reason why AOT-compiled
> languages like Go with giant binaries are gaining favor
> nowadays.


Is it not possible to go the path that Mark suggests by compiling a big
Perl binary with the additional modules?
0
xsawyerx
12/28/2019 7:48:04 PM
Sawyer X <xsawyerx@gmail.com> wrote:
> 
> On 12/17/19 9:18 AM, Eric Wong wrote:
> >> On Mon, Dec 16, 2019 at 2:51 PM Eric Wong <p5p@yhbt.net> wrote:
> >>
> >>> Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
> >>> but I figure it'll be useful to support new functionality...
> > Hi Mark, Dan; thanks for the responses so far.
> >
> > Some background here: most users of my software get Perl from
> > their GNU/Linux distros or BSD ports.  And a big reason why I
> > choose Perl is that it's widely available out-of-the-box and
> > users typically don't need to install extra dependencies.
> >
> > CPAN modules which aren't packaged by the distro are extra
> > overhead for them to learn, download and install; and they
> > already complain about dependencies.  Most of them are not Perl
> > hackers.  Even getting them to consider using software written
> > in Perl these days is a challenge :<
> >
> > Mark Overmeer <mark@overmeer.net> wrote:
> >> There are about 1800 functions in the 2008 release.  POSIX.pm
> >> and CORE together support about 100 (but also not really strict) 
> >>
> >> There are some modules on CPAN which implement some additional
> >> functions.  A few dozen.
> > Right, some are abandoned and most of those are not available
> > from distros.
> >
> >> I started POSIX::1003 which tries to provide all functions... as
> >> pure as feasible.  But... I need to rework it into a more flexible
> >> installation structure using Config::AutoConf.  Help welcome ;-)
> >> When I get help, I will increase its priority.
> > Any reason you choose to work on a new module rather than
> > improving the existing POSIX one?  I certainly don't see the
> > need for all functions in POSIX, especially the ones which
> > are made redundant by CORE functions.
> >
> > Dan Book <grinnz@gmail.com> wrote:
> >> If it's at all helpful, it's trivial to write a FFI::Platypus wrapper of
> >> POSIX functions available on the system, such as I did in Unix::Groups::FFI.
> > Yup.  I favor Inline(::C) since it's more widely-packaged
> > (for example, perl-Inline is in CentOS/RHEL 7.x and AFAIK
> > Platypus is not).
> >
> > I may even favor syscall() for popular platforms since there's
> > no extra .so loading and no need to have a compiler installed.
> >
> > But I consider those dependencies interim solutions because of
> > the extra installation burden it places on users (and distro
> > maintainers).  And this probably a big reason why AOT-compiled
> > languages like Go with giant binaries are gaining favor
> > nowadays.
> 
> 
> Is it not possible to go the path that Mark suggests by compiling a big
> Perl binary with the additional modules?

I'm trying to reduce the long-term maintenance and update
overhead, so I wouldn't even consider asking users to
build/maintain their own Perl install if they aren't comfortable
with dependencies which require going outside the distro to
CPAN.

Again, getting users to even consider software written in Perl
is a challenge these days.  The fact that Perl5 is packaged, and
often installed-by-default on Linux/*BSD distros is something I
want to take advantage of in promoting its use.
0
p5p
12/28/2019 8:51:14 PM
* Eric Wong (p5p@yhbt.net) [191228 20:51]:
> > Is it not possible to go the path that Mark suggests by compiling a big
> > Perl binary with the additional modules?
> 
> I'm trying to reduce the long-term maintenance and update
> overhead, so I wouldn't even consider asking users to
> build/maintain their own Perl install if they aren't comfortable
> with dependencies which require going outside the distro to
> CPAN.

Packagers of distributions are very helpful, so they will contribute
to maintain such a large package... at least (and most important)
report discovered issues.  Rarely used modules (which happens when
you fragment the solution) however, will simply never get packaged
in the first place.  Issues will be discovered much later.

In manual pages, you can only find the current status of you OS libraries.
When you write code, you have to have some backwards compatibility. I used
the output of cpantesters to collect OS differences.
See http://posix.cpan6.net/show/latest/fdio.html
-- 
               MarkOv

------------------------------------------------------------------------
       Mark Overmeer MSc                                MARKOV Solutions
       Mark@Overmeer.net                          solutions@overmeer.net
http://Mark.Overmeer.net                   http://solutions.overmeer.net
0
mark
12/29/2019 10:48:32 AM
On 12/28/19 10:51 PM, Eric Wong wrote:
> Sawyer X <xsawyerx@gmail.com> wrote:
>> On 12/17/19 9:18 AM, Eric Wong wrote:
>>>> On Mon, Dec 16, 2019 at 2:51 PM Eric Wong <p5p@yhbt.net> wrote:
>>>>
>>>>> Hello all, I noticed POSIX.pod states it's for IEEE Std 1003.1,
>>>>> but I figure it'll be useful to support new functionality...
>>> Hi Mark, Dan; thanks for the responses so far.
>>>
>>> Some background here: most users of my software get Perl from
>>> their GNU/Linux distros or BSD ports.  And a big reason why I
>>> choose Perl is that it's widely available out-of-the-box and
>>> users typically don't need to install extra dependencies.
>>>
>>> CPAN modules which aren't packaged by the distro are extra
>>> overhead for them to learn, download and install; and they
>>> already complain about dependencies.  Most of them are not Perl
>>> hackers.  Even getting them to consider using software written
>>> in Perl these days is a challenge :<
>>>
>>> Mark Overmeer <mark@overmeer.net> wrote:
>>>> There are about 1800 functions in the 2008 release.  POSIX.pm
>>>> and CORE together support about 100 (but also not really strict) 
>>>>
>>>> There are some modules on CPAN which implement some additional
>>>> functions.  A few dozen.
>>> Right, some are abandoned and most of those are not available
>>> from distros.
>>>
>>>> I started POSIX::1003 which tries to provide all functions... as
>>>> pure as feasible.  But... I need to rework it into a more flexible
>>>> installation structure using Config::AutoConf.  Help welcome ;-)
>>>> When I get help, I will increase its priority.
>>> Any reason you choose to work on a new module rather than
>>> improving the existing POSIX one?  I certainly don't see the
>>> need for all functions in POSIX, especially the ones which
>>> are made redundant by CORE functions.
>>>
>>> Dan Book <grinnz@gmail.com> wrote:
>>>> If it's at all helpful, it's trivial to write a FFI::Platypus wrapper of
>>>> POSIX functions available on the system, such as I did in Unix::Groups::FFI.
>>> Yup.  I favor Inline(::C) since it's more widely-packaged
>>> (for example, perl-Inline is in CentOS/RHEL 7.x and AFAIK
>>> Platypus is not).
>>>
>>> I may even favor syscall() for popular platforms since there's
>>> no extra .so loading and no need to have a compiler installed.
>>>
>>> But I consider those dependencies interim solutions because of
>>> the extra installation burden it places on users (and distro
>>> maintainers).  And this probably a big reason why AOT-compiled
>>> languages like Go with giant binaries are gaining favor
>>> nowadays.
>>
>> Is it not possible to go the path that Mark suggests by compiling a big
>> Perl binary with the additional modules?
> I'm trying to reduce the long-term maintenance and update
> overhead, so I wouldn't even consider asking users to
> build/maintain their own Perl install if they aren't comfortable
> with dependencies which require going outside the distro to
> CPAN.


This might be out of your scope, but I'll offer this as another option:


I had a project that was self-hosting Perl setup with dependencies
without packaging. The way it worked was:


* A small script that used Seacan[1] to setup a self-contained my
project code, a perl binary, and all dependencies using local::lib.

* The script was then FatPacked[2] so it itself won't require anything
other than perl > v5.10 to generate the full instance (step 1)


When we wanted to build a new version of it, the FatPacked script was
run, created the whole environment using Seacan, and then users could
just run it with app.pl that Seacan created.


I can share with you privately the work, if it helps.


[1] https://github.com/gugod/Seacan

[2] https://metacpan.org/pod/distribution/App-FatPacker/bin/fatpack
0
xsawyerx
12/29/2019 3:31:55 PM
Reply: