Managing Deployment of Perl Modules
In reaching for the goal of fully automated deployment, I frequently find myself wishing that various languages had better support for package/module/whatever management.
Each language has its own tool(s?) for installing modules that usually downloads the source from either a canonical source or a mirrorlist.
But I want to use the OS/distribution's native tools for package management that are best suited for upgrade management, dependency resolution, etc. In the world of redhat-based distributions, that means a local yum repository of locally built rpms. (The debian world has dh-make-perl which helps with this problem; but debian is not my world, so I'll leave that alone for now.)
Using RPMs for perl modules
There are various methods for this, but basically it goes like this:
- Generate a first draft of a spec file from CPAN or the module's tarball/META.yml
- Fix the specfile
- Build the (s)rpms that you want
- Put them in a local yum repository
Here are some comments from the Fedora Perl SIG: https://fedoraproject.org/wiki/Packaging:Perl
cpanspec
cpanspec is used by the Fedora distribution's perl module package maintainers as at least a starting point to automatically generate rpm spec files from CPAN. Unfortunately, a cpanspec package it wasn't just available for the CentOS version that I was using, so I had to get it myself (it's just a perl script, but I had to install some dependencies by hand/cpanm). To create a package for the perl module Foo::Bar, I use it something like this:
cpanspec --license --packager 'Duncan Hutty <dhutty@allgoodbits.org>' --verbose --follow Foo::Bar
Fixing the specfile is usually a twofold task: ensure that all installed files are packaged and correct the Requires.
Ensure that all files that will be installed are packaged (listed in %files)
ensuring that the %files section of the specfile is correct, to avoid an error like Installed (but unpackaged) file(s) found:. For example, for Env::Path, I had 2 errors:
Installed (but unpackaged) file(s) found: /usr/bin/envpath /usr/share/man/man1/envpath.1.gz
which meant I had to add 2 lines to %files:
%{_bindir}/envpath %{_mandir}/man1/*
Filter Requires
This snippet can go immediately after the metadata at the beginning of the specfile, before even the %description. Without it, when I tried to actually install the generated rpms, yum failed with failed Require of /usr/local/bin/perl:
%{?filter_setup: %{?perl_default_filter} }
Here is an example of real filtering for the perl module Clipboard so that it doesn't require/provide Win32/Mac specific modules:
%filter_from_provides /Clipboard::Win32/d %filter_from_provides /Clipboard::MacPasteboard/d %filter_from_requires /Win32::Clipboard/d %filter_from_requires /Mac::Pasteboard/d %filter_setup
cpan2dist
Another possibility here is: cpan2dist. I list this because it's more easily available and sometimes it's been good enough for me, but as I've needed more and more perl modules, I've found that cpanspec has been working better for me.
cpan2dist --format CPANPLUS::Dist::RPM \ --default-ignorelist \ --default-banlist \ --ignore Devel::StackTrace \ --ignore Data::Dumper \ <module to build>
Building the RPM
rpmbuild -ba perl-Foo-Bar.spec
Add the rpm to a local repository
for r in `ls ~/rpmbuild/RPMS`; do createrepo -d ~/rpmbuild/RPMS/${r} done
And then ensure that your ~/rpmbuild/RPMS is served by your installation server as the rootdir of your local yum repo
Alternative Approach with local::lib
There is an alternative approach that I have found works for certain situations. It relies upon cpanm to install to local::lib and puppet to manage which locallib directories get installed on individual machines. Using the distributions package management is strongly preferred, because it means you have a single database (per system) that contains information about what software is installed, and because installing binary packages, you're not building on every system. But sometimes, you just need this.
local::lib
local::lib provides a local lib directory that we can fill with the exact versions of various perl modules that we want to reduce the chance of problems from conflicts and/or dependency hell.
- The only constraint imposed on the developer is the requirement for a use pragma according to which version of the module library is desired::
- use local::lib '/usr/local/perl/best' use local::lib '/usr/local/perl/35
Installing into local::lib with cpanm
Trivial:
cpanm -L <directory> <module to install>
Getting local::lib dirs in place
In this case, I'm using puppet for configuration management, so for each application server, puppet knows which perl module directories are required. I also have puppet create a symlink to the newest:
ls -l /agb/perl total 0 drwxr-xr-x 2 root root 40 Jan 9 10:19 12 drwxr-xr-x 2 root root 40 Jan 9 10:19 3 drwxr-xr-x 2 root root 40 Jan 9 10:19 5 drwxr-xr-x 2 root root 40 Jan 9 10:19 8 lrwxrwxrwx 1 root root 2 Jan 9 10:19 best -> 12
- This means that on a development machine, devs/apps can ask for the latest versions that I have published and know that they're developing against the latest edge, but when we come to deploy, we can lock to a particular set of module versions for stability. Here's how to ask::
- use local::lib '/agb/perl/best' use local::lib '/agb/perl/3'
Multiple versions of perl
In order to take care of the possible requirement for multiple versions of perl, we can push the above structure down one level and use some other symlink magic for versions numbers:
ls -l /agb/perl total 0 drwxr-xr-x 2 root root 40 Jan 9 10:19 5.10.1 drwxr-xr-x 2 root root 40 Jan 9 10:19 5.12.4 lrwxrwxrwx 1 root root 2 Jan 9 10:19 best -> 5.12.4 ls -l /agb/perl/5.10.1 total 0 drwxr-xr-x 2 root root 40 Jan 9 10:19 1 drwxr-xr-x 2 root root 40 Jan 9 10:19 2 drwxr-xr-x 2 root root 40 Jan 9 10:19 4 drwxr-xr-x 2 root root 40 Jan 9 10:19 6 lrwxrwxrwx 1 root root 2 Jan 9 10:19 best -> 6 ls -l /agb/perl/5.12.4 total 0 drwxr-xr-x 2 root root 40 Jan 9 10:19 12 drwxr-xr-x 2 root root 40 Jan 9 10:19 3 drwxr-xr-x 2 root root 40 Jan 9 10:19 5 drwxr-xr-x 2 root root 40 Jan 9 10:19 8 lrwxrwxrwx 1 root root 2 Jan 9 10:19 best -> 12
- If you're going to take this approach, here's how to ask::
- use local::lib '/agb/perl/best/best' use local::lib '/agb/perl/5.10.1/2'
Alternative Approach with perlbrew
The following approach seemed to be the best solution for one situation:
- use perlbrew to install a new, separate perl into /opt
- use App::cpanminus (aka cpanm) to install dependency modules into it
- install the desired application also into /opt taking advantage of the perlbrewed version of perl
- package the whole lot as a mega, self-contained rpm.
Example specfile of the perlbrew/megarpm approach for RT:
Name: rt Version: 4.2.0 Release: 1%{?dist} Summary: Request Tracker, the Mega-rpm, including its own perl Group: Applications/Internet License: GPLv2+ URL: http://www.bestpractical.com/rt BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX) BuildRequires: perlbrew BuildRequires: expat-devel BuildRequires: gd-devel BuildRequires: mysql-devel BuildRequires: openssl-devel Requires: mysql Requires: gd Requires: gnupg2 Requires: graphviz Requires: openldap Requires: openssl AutoReqProv: no %global RT_PREFIX /opt/rt4 %global PERL_PREFIX /opt/perl5 # NOTICE: you'll need to modify the __os_install_post macro that you can see # in your rpm --showrc output, so override it in e.g. ~/.rpmmacros. It # *shouldn't* be here because specfiles don't support multiline macro defs. # Just remove line %{_rpmconfigdir}/brp-strip-static-archive %description RT is an enterprise-grade ticketing system which enables a group of people to intelligently and efficiently manage tasks, issues, and requests submitted by a community of users. This build is with a custom perl5 install to keep it entirely separate from vendor perl. %prep %build %install rm -rf %{buildroot} echo "Copy complete perl5 base directory to buildroot" mkdir -p %{buildroot}/%{PERL_PREFIX} cp -Rp %{PERL_PREFIX}/* %{buildroot}/%{PERL_PREFIX}/ echo "Copy complete rt4 base directory to buildroot" mkdir -p %{buildroot}/%{RT_PREFIX} cp -Rp %{RT_PREFIX}/* %{buildroot}/%{RT_PREFIX}/ %clean rm -rf %{buildroot} %files #custom perl5 %defattr(644,root,root) %attr(755,root,root) %{PERL_PREFIX}/bin/* %{PERL_PREFIX}/etc/* %{PERL_PREFIX}/perls/* %{PERL_PREFIX}/Conf.pm %dir %attr(755, root, root) %{PERL_PREFIX}/bin %dir %attr(755, root, root) %{PERL_PREFIX}/etc %dir %attr(755, root, root) %{PERL_PREFIX}/perls #rt stuff #%config{noreplace} %{_sysconfdir}/init.d/rt #%config{noreplace} /opt/rt4/etc/RT_SiteConfig.pm %attr(755,root,root) %{RT_PREFIX}/bin/* %attr(755,root,root) %{RT_PREFIX}/sbin/* %attr(0644,apache,apache) %{RT_PREFIX}/etc/* %{RT_PREFIX}/lib/* %{RT_PREFIX}/share/html/* %{RT_PREFIX}/share/po/* %{RT_PREFIX}/share/fonts/* %attr(0755,apache,apache) %{RT_PREFIX}/var/log %attr(0770,apache,apache) %{RT_PREFIX}/var/mason_data %attr(0770,apache,apache) %{RT_PREFIX}/var/session_data %{RT_PREFIX}/local/* %changelog * Fri Nov 8 2013 Duncan Hutty <dhutty@allgoodbits.org>