Actions: | Security

AllGoodBits.org

Navigation: Home | Services | Tools | Articles | Other

Managing Deployment of Perl Modules

In reaching for the goal of fully automated deployment, I frequently find myself wishing that various languages had better support for package/module/whatever management.

Each language has its own tool(s?) for installing modules that usually downloads the source from either a canonical source or a mirrorlist.

But I want to use the OS/distribution's native tools for package management that are best suited for upgrade management, dependency resolution, etc. In the world of redhat-based distributions, that means a local yum repository of locally built rpms. (The debian world has dh-make-perl which helps with this problem; but debian is not my world, so I'll leave that alone for now.)

Using RPMs for perl modules

There are various methods for this, but basically it goes like this:

Here are some comments from the Fedora Perl SIG: https://fedoraproject.org/wiki/Packaging:Perl

cpanspec

cpanspec is used by the Fedora distribution's perl module package maintainers as at least a starting point to automatically generate rpm spec files from CPAN. Unfortunately, a cpanspec package it wasn't just available for the CentOS version that I was using, so I had to get it myself (it's just a perl script, but I had to install some dependencies by hand/cpanm). To create a package for the perl module Foo::Bar, I use it something like this:

cpanspec --license --packager 'Duncan Hutty <dhutty@allgoodbits.org>'  --verbose   --follow Foo::Bar

Fixing the specfile is usually a twofold task: ensure that all installed files are packaged and correct the Requires.

Ensure that all files that will be installed are packaged (listed in %files)

ensuring that the %files section of the specfile is correct, to avoid an error like Installed (but unpackaged) file(s) found:. For example, for Env::Path, I had 2 errors:

Installed (but unpackaged) file(s) found:
/usr/bin/envpath
/usr/share/man/man1/envpath.1.gz

which meant I had to add 2 lines to %files:

%{_bindir}/envpath
%{_mandir}/man1/*
Filter Requires

This snippet can go immediately after the metadata at the beginning of the specfile, before even the %description. Without it, when I tried to actually install the generated rpms, yum failed with failed Require of /usr/local/bin/perl:

%{?filter_setup:
%{?perl_default_filter}
}

Here is an example of real filtering for the perl module Clipboard so that it doesn't require/provide Win32/Mac specific modules:

%filter_from_provides /Clipboard::Win32/d
%filter_from_provides /Clipboard::MacPasteboard/d
%filter_from_requires /Win32::Clipboard/d
%filter_from_requires /Mac::Pasteboard/d
%filter_setup

cpan2dist

Another possibility here is: cpan2dist. I list this because it's more easily available and sometimes it's been good enough for me, but as I've needed more and more perl modules, I've found that cpanspec has been working better for me.

cpan2dist --format CPANPLUS::Dist::RPM \
          --default-ignorelist \
          --default-banlist \
          --ignore Devel::StackTrace \
          --ignore Data::Dumper \
          <module to build>

Building the RPM

rpmbuild -ba perl-Foo-Bar.spec

Add the rpm to a local repository

for r in `ls ~/rpmbuild/RPMS`;
  do
    createrepo -d ~/rpmbuild/RPMS/${r}
  done

And then ensure that your ~/rpmbuild/RPMS is served by your installation server as the rootdir of your local yum repo

Alternative Approach with local::lib

There is an alternative approach that I have found works for certain situations. It relies upon cpanm to install to local::lib and puppet to manage which locallib directories get installed on individual machines. Using the distributions package management is strongly preferred, because it means you have a single database (per system) that contains information about what software is installed, and because installing binary packages, you're not building on every system. But sometimes, you just need this.

local::lib

local::lib provides a local lib directory that we can fill with the exact versions of various perl modules that we want to reduce the chance of problems from conflicts and/or dependency hell.

The only constraint imposed on the developer is the requirement for a use pragma according to which version of the module library is desired::
use local::lib '/usr/local/perl/best' use local::lib '/usr/local/perl/35

Installing into local::lib with cpanm

Trivial:

cpanm -L <directory> <module to install>

Getting local::lib dirs in place

In this case, I'm using puppet for configuration management, so for each application server, puppet knows which perl module directories are required. I also have puppet create a symlink to the newest:

ls -l /agb/perl
total 0
drwxr-xr-x 2 root root 40 Jan  9 10:19 12
drwxr-xr-x 2 root root 40 Jan  9 10:19 3
drwxr-xr-x 2 root root 40 Jan  9 10:19 5
drwxr-xr-x 2 root root 40 Jan  9 10:19 8
lrwxrwxrwx 1 root root  2 Jan  9 10:19 best -> 12
This means that on a development machine, devs/apps can ask for the latest versions that I have published and know that they're developing against the latest edge, but when we come to deploy, we can lock to a particular set of module versions for stability. Here's how to ask::
use local::lib '/agb/perl/best' use local::lib '/agb/perl/3'

Multiple versions of perl

In order to take care of the possible requirement for multiple versions of perl, we can push the above structure down one level and use some other symlink magic for versions numbers:

ls -l /agb/perl
total 0
   drwxr-xr-x 2 root root 40 Jan  9 10:19 5.10.1
   drwxr-xr-x 2 root root 40 Jan  9 10:19 5.12.4
   lrwxrwxrwx 1 root root  2 Jan  9 10:19 best -> 5.12.4

ls -l /agb/perl/5.10.1
total 0
   drwxr-xr-x 2 root root 40 Jan  9 10:19 1
   drwxr-xr-x 2 root root 40 Jan  9 10:19 2
   drwxr-xr-x 2 root root 40 Jan  9 10:19 4
   drwxr-xr-x 2 root root 40 Jan  9 10:19 6
   lrwxrwxrwx 1 root root  2 Jan  9 10:19 best -> 6

ls -l /agb/perl/5.12.4
total 0
   drwxr-xr-x 2 root root 40 Jan  9 10:19 12
   drwxr-xr-x 2 root root 40 Jan  9 10:19 3
   drwxr-xr-x 2 root root 40 Jan  9 10:19 5
   drwxr-xr-x 2 root root 40 Jan  9 10:19 8
   lrwxrwxrwx 1 root root  2 Jan  9 10:19 best -> 12
If you're going to take this approach, here's how to ask::
use local::lib '/agb/perl/best/best' use local::lib '/agb/perl/5.10.1/2'

Alternative Approach with perlbrew

The following approach seemed to be the best solution for one situation:

Example specfile of the perlbrew/megarpm approach for RT:

Name:    rt
Version: 4.2.0
Release: 1%{?dist}
Summary: Request Tracker, the Mega-rpm, including its own perl

Group:   Applications/Internet
License: GPLv2+
URL:     http://www.bestpractical.com/rt

BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)

BuildRequires: perlbrew
BuildRequires: expat-devel
BuildRequires: gd-devel
BuildRequires: mysql-devel
BuildRequires: openssl-devel
Requires: mysql
Requires: gd
Requires: gnupg2
Requires: graphviz
Requires: openldap
Requires: openssl

AutoReqProv: no

%global RT_PREFIX /opt/rt4
%global PERL_PREFIX /opt/perl5

# NOTICE: you'll need to modify the __os_install_post macro that you can see
# in your rpm --showrc output, so override it in e.g. ~/.rpmmacros. It
# *shouldn't* be here because specfiles don't support multiline macro defs.
# Just remove line %{_rpmconfigdir}/brp-strip-static-archive

%description
RT is an enterprise-grade ticketing system which enables a group of people
to intelligently and efficiently manage tasks, issues, and requests submitted
by a community of users.

This build is with a custom perl5 install to keep it entirely separate from
vendor perl.

%prep

%build

%install

rm -rf %{buildroot}
echo "Copy complete perl5 base directory to buildroot"
mkdir -p %{buildroot}/%{PERL_PREFIX}
cp -Rp %{PERL_PREFIX}/* %{buildroot}/%{PERL_PREFIX}/

echo "Copy complete rt4 base directory to buildroot"
mkdir -p %{buildroot}/%{RT_PREFIX}
cp -Rp %{RT_PREFIX}/* %{buildroot}/%{RT_PREFIX}/

%clean
rm -rf %{buildroot}


%files
#custom perl5
%defattr(644,root,root)
%attr(755,root,root) %{PERL_PREFIX}/bin/*
%{PERL_PREFIX}/etc/*
%{PERL_PREFIX}/perls/*
%{PERL_PREFIX}/Conf.pm
%dir %attr(755, root, root) %{PERL_PREFIX}/bin
%dir %attr(755, root, root) %{PERL_PREFIX}/etc
%dir %attr(755, root, root) %{PERL_PREFIX}/perls

#rt stuff

#%config{noreplace} %{_sysconfdir}/init.d/rt
#%config{noreplace} /opt/rt4/etc/RT_SiteConfig.pm

%attr(755,root,root) %{RT_PREFIX}/bin/*
%attr(755,root,root) %{RT_PREFIX}/sbin/*
%attr(0644,apache,apache) %{RT_PREFIX}/etc/*
%{RT_PREFIX}/lib/*
%{RT_PREFIX}/share/html/*
%{RT_PREFIX}/share/po/*
%{RT_PREFIX}/share/fonts/*
%attr(0755,apache,apache) %{RT_PREFIX}/var/log
%attr(0770,apache,apache) %{RT_PREFIX}/var/mason_data
%attr(0770,apache,apache) %{RT_PREFIX}/var/session_data
%{RT_PREFIX}/local/*

%changelog
* Fri Nov 8 2013 Duncan Hutty <dhutty@allgoodbits.org>