And then, when we write an LWP script to check our bank account, we hardcode the password. That's ... stupid. But what alternatives are there? There's File::KeePass, (as recently featured on PerlTricks.com) - but you still have to supply it with a password to unlock the database, and that would expose all your passwords to a hypothetical laptop thief. That's worse!
You can just prompt for a password every time the script runs - but what if I want a scheduled script that checks my bank balance every ten minutes and sounds a siren if things go wrong? More seriously, prompting for a password defeats the purpose of any automation, so it's only a partial answer.
If only there were a way to start the KeePass utility with a password once, as intended, then have it run in the background to supply automated scripts with passwords when needed! Turns out this is actually pretty easy, with a nifty plugin called KeePassRest, and the Perl module WWW::KeePassRest.
KeePassRest was developed by the makers of SmartFTP to store FTP server passwords securely. (Did you know that FileZilla keeps your passwords in a plaintext XML file? No? Surprise!) In other words, it was developed precisely for this situation. Once installed, it listens on port 12984 of your local machine when KeePass is started, and fields REST requests against the current database in JSON format. The API it exposes doesn't do everything File::KeePass can do - most particularly it doesn't do anything with groups - but if you use KeePass to organize your keys, you can use the API to retrieve passwords for automation without any further ado. It's an excellent part of a sensible automation security strategy.
The POD in WWW::KeePassRest includes example code showing how to use the API, but this article presents some background on the technical issues, because frankly, anything to do with security is fraught with confusion as far as I'm concerned. So I wanted to document a little of what I learned while writing WWW::KeePassRest, if only to have it all in one place next time I want to do something security-related. And there were some other neat little tips and tricks I learned along the way, as well.
The HTTPS protocol used to talk to KeePassRest uses TLS (transport layer security) to encrypt everything on the wire, and TLS uses X.509 certificates to do that. So why do we always call everything SSL? It's easy to remember, that's why - and the technical jargon for anything remotely related to cryptography is an impenetrable morass of legalism and terminology that I, for one, find incredibly daunting.
So it was with immense relief that I found this awesome SSL Survival Guide, which explains everything with a touch of humor entirely absent from any other security-related material I have ever seen. Read it. Then read it a couple more times. Eventually some of it will sink in. (At least that's what I've been telling myself.) Seriously - I've been working intermittently with this stuff since the 90's and I just can't remember any of it between exposures.
At any rate, the point is that to connect to an HTTPS service, you've got to have an X.509 certificate. When I first started figuring out KeePassRest, I was under the mistaken impression that - since HTTPS "runs on SSL", these were the same as the keys I use for password-free authentication to SSL servers. They are not. They're used more or less the same way, but we're talking about different protocols entirely. (And Net::SSL failed silently, leading to several hours of my wondering why a "certificate" that Net::SSL seemed to like wasn't being accepted by KeePassRest...)
To talk to KeePassRest, you need an X.509 certificate and key in two files conventionally stored with names ending in ".pem". The certificate is the public part that will be sent to the server, and the key is the private part you'll use (well - LWP will use ... well - Net::SSL will use, well no, actually Crypt::SSLeay will use, it's a tall stack) to encrypt messages going out.
You can create your own key and certificate with OpenSSL following the instructions in the survival guide for creating a single self-signed certificate, but be warned - lots of different systems install OpenSSL, and not all of them will actually work. In my case, I had to juggle my path to be sure that an OpenSSL that Strawberry's installer had left in place wasn't masking the Cygwin version that should have been called - when called in the Strawberry environment, there is apparently a virtual drive Z: that is used to store the configuration file, and so it fails when called from the command line.
In the WWW::KeePassRest module, I've bundled perfectly good self-signed cert and key files for use by the module, so you don't even need to worry about any of this configuration to use the module. (And more saliently, I don't have to forget where I stored the cert.) I generated those with this:
C:\projects\keepass\cert>openssl req -x509 -nodes -days 10000 -newkey rsa:2048 -keyout wwwkprkey.pem -out wwwkprcert.pem Generating a 2048 bit RSA private key ................................................................................ ...................................................+++ ......................................+++ writing new private key to 'wwwkprkey.pem' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:. State or Province Name (full name) [Some-State]:. Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]:CPAN Organizational Unit Name (eg, section) []: Common Name (eg, YOUR name) []:WWW::KeePassRest Email Address []:
You can easily generate your own if you don't want to use the default key/certificate.
The other place I shot myself in the foot during this whole venture was not quite clearly understanding all the different Perl modules interacting and how each is parameterized. The LWP user agent itself configures the SSL connection based on the values passed in the ssl_opts method, but passes that configuration to the SSL module actually in use (either IO::Socket::SSL or Net::SSL; in my case, Net::SSL).
If you explicitly use Net::SSL, though, then you can use its own configuration mechanism, which consists of reading environment variables. As I was flailing around trying to figure out why SSL wasn't working, I managed to mix and match these, except for the match part, and ended up entirely confused. (The environment variables are actually used by Crypt::SSLeay down there at the bottom of the stack; I believe they're probably overwritten by LWP::UserAgent if Net::SSL isn't explicitly loaded, but I haven't looked at the code to verify this.)
The most interesting of those is actually HTTPS_DEBUG; if set to 1, that will give you very verbose step-by-step information about what is happening and what might be going wrong in this many-layered process. I found this invaluable in understanding the many things I screwed up.
But normally, you don't load Net::SSL explicitly, and you don't need to mess with the environment variables.
I mentioned that the WWW::KeePassRest module provides its own certificate so you don't have to worry about installing one. This wasn't the first time I'd thought about how nice it would be to provide non-module data files during installation from CPAN, but it is the first time I actually figured out how it's done, so I thought I'd mention it here before I get down to the business of protyping.
It's actually pretty straightforward. Perl provides an 'auto' facility that can be used for the installation of static files either per distribution or per module, and the File::ShareDir module can find them and return the full directory (see the module documentation for details). If you use Ext::MakeMaker for your builds, like I do (I know, I know, but I'm familiar with it), then the File::ShareDir::Install module will augment your build process with just a couple of lines to install your static files for you.
The nice thing is that they're also put into their proper slots for testing. Slick! I love these little extras in the Perl module/CPAN ecosystem.
Finally, you'll notice if you run the sample script that the first time a certificate/key is used to access KeePassRest, a security dialog pops up like this:
If you click Yes, you won't see that box again - as long as you save your KeePass database before ending the session. That's because the KeePassRest plugin saves all the keys in the extra string values (on the "Advanced" tab of the key information) of a key named "KeePassRest Settings", stored by default in the root group of your database. Honestly, KeePass would be a pretty great way to store any kind of sensitive configuration information on Windows (and of course there are work-alikes on *nix systems that even use the same database format).
So now that we understand the technical background, let's look at the first prototype script I wrote to talk to KeePassRest at the LWP level. What this script does is pretty simple - it asks KeePassRest to search for keys stored with "GitHub" in their titles, then retrieves each of those keys and shows detailed information about them. On my machine, that returns and displays two different keys, so it was a good test case. Your mileage will obviously vary, depending on what you have in your KeePass database.
01 use LWP; 02 use HTTP::Request; 03 use Data::Dumper; 04 use JSON; 05 06 my $ua = LWP::UserAgent->new(); 07 $ua->ssl_opts ( 08 SSL_version => 'SSLv3', 09 verify_hostname => 0, 10 SSL_cert_file => 'cert/wwwkprcert.pem', 11 SSL_key_file => 'cert/wwwkprkey.pem', 12 ); 13 14 my $j = JSON->new->utf8; 15 my $req; 16 my $res; 17 18 $req = HTTP::Request->new ( POST => 'https://localhost:12984/keepass/entry/search', 19 ['Content-Type' => 'application/json'], 20 '{"SearchString":"Github","SearchInTitles":"true"}' ); 21 22 $res = $ua->request($req); 23 warn $@ if $@; 24 25 my $c = $j->decode($res->content); 26 print Dumper($c); 27 foreach my $uuid (@$c) { 28 $req = HTTP::Request->new (GET => "https://localhost:12984/keepass/entry/$uuid" ); 29 $res = $ua->request ($req); 30 my $entry = $j->decode($res->content); 31 print Dumper($entry); 32 }
On my machine, this returns the following (except obviously the passwords are not all x's):
C:\projects\keepass>perl keepass-raw.pl
$VAR1 = [
'7F7DE15ADED7F74BB311C94932F7DE0D',
'B98ED52729AFCC43970CE04FE6DEA636'
];
$VAR1 = {
'UserName' => 'x5n',
'URL' => '',
'Password' => 'xxxxxxxxxxxxxxx',
'Notes' => 'Email address vivtek@gmail.com',
'Title' => 'Github @ x5n'
};
$VAR1 = {
'UserName' => 'michael@vivtek.com',
'URL' => 'https://github.com/signup/free',
'Password' => 'xxxxxxxxxxxxxxx',
'Notes' => "This is a UTF-8 t\x{e9}st.",
'Title' => 'Github @ Vivtek'
};
To exercise the rest of the KeePassRest API, I wanted to add calls to create, update, and delete a key - but all the typing to do that started to get boring. I abstracted out the JSON calls so I could concentrate on just the more KeePassRest-specific logic (and also type a little less). I put that stuff into a new module I've released to CPAN as WWW::JSONAPI; I'll link to the new article on that as soon as it's written. (I will note, however, that there is already a perfectly good CPAN module called WWW::JSON that may very well suit your needs better, as it is far more mature.)
I defined five different methods against a JSON API connection opened with this module; these are json_POST_json (posts JSON, then expects a JSON return), json_POST_string (posts JSON, then expects a string in return), json_PUT_string, GET_json (doesn't post anything, gets JSON back), and finally DELETE_string (doesn't post anything, gets a string back). (I also tossed in some error handling so it doesn't clutter up our code - it will croak gracefully if KeePass isn't started and thus not accepting connections, for example.)
When we abstract that out and add the new logic, we get my second prototype. Here, note that lines 1-18 are just the translation of lines 1-32 of the first prototype, so I saved that much typing, anyway.
01 use strict; use warnings; 02 use lib './lib'; 03 use WWW::JSONAPI; 04 use Data::Dumper; 05 06 my $json = WWW::JSONAPI->new(cert_file => 'cert/wwwkprcert.pem', 07 key_file => 'cert/wwwkprkey.pem', 08 base_url => 'https://localhost:12984/keepass/'); 09 10 # Search for keys with "Github" in the title 11 my $c = $json->json_POST_json ('entry/search', { SearchString => 'Github', 12 SearchInTitles => 'true' } ); 13 print Dumper($c); 14 15 foreach my $uuid (@$c) { 16 my $entry = $json->GET_json ("entry/$uuid"); 17 print Dumper($entry); 18 } 19 20 # Get a list of existing keys that already have our test URL. 21 my $list = $json->json_POST_json ('entry/search', { SearchString => 'http://test.cpan.org/wwwkpr', SearchInUrls => 'true'} ); 22 23 # Add a test key - in contrast with the spec, this returns "true" if successful, not the UUID. 24 my $ret = $json->json_POST_string ('entry', { GroupName => 'test', 25 Entry => { Title => 'wwwkpr test', 26 Password => 'test', 27 URL => 'http://test.cpan.org/wwwkpr' }}); 28 print "$ret\n"; 29 30 $c = $json->json_POST_json ('entry/search', { SearchString => 'http://test.cpan.org/wwwkpr', SearchInUrls => 'true'} ); 31 my $newkey = ''; 32 foreach my $check (@$c) { 33 if (not grep { $_ eq $check } @$list) { 34 $newkey = $check; 35 last; 36 } 37 } 38 print "$newkey\n"; 39 40 my $entry = $json->GET_json ("entry/$newkey"); 41 print Dumper($entry); 42 43 $entry->{'test-string'} = $newkey; 44 print $json->json_PUT_string ("entry/$newkey", $entry) . "\n"; 45 46 print $json->DELETE_string ("entry/$newkey") . "\n";
This one returns the same information as above, then continues:
true
3422B28F08767C4D9B352BEF532B7215
$VAR1 = {
'URL' => 'http://test.cpan.org/wwwkpr',
'Password' => 'test',
'Title' => 'wwwkpr test'
};
true
true
I know it's shocking, but that prototype just covered the entire API that KeePassRest exposes. The only thing it didn't do was exercise all the different options, but basically you've got CRUD for entries, and search for entries (plus incidental creation of a group if the group you specify when creating a key doesn't exist). The folks at SmartFTP have documented the API here, but I found that coverage really thin while I was trying to figure this out. As I note in the second prototype, their API actually contains an error; instead of returning the UUID key for the entry created by Create, their API simply returns 'true' if successful.
The WWW::KeePassRest module actually implements two levels of API: the first is a simple wrapping of the REST API provided by the plugin, and consists of 'create', 'get', 'update', 'delete', and 'search' that do exactly what you'd expect.
Then I added some conveniences: the get method, in addition to the UUID of the entry to retrieve, can also take a list of fields to extract. Then, instead of getting a hashref back, you get a list of the values for those fields - so you can pass the UserName and Password fields directly to your authentication call in LWP.
Two methods get_by_title and get_by_url do exactly the same thing as vanilla get, except that instead of a UUID you can pass the title or URL of your desired entry. If there are collisions (multiple entries are returned from the search), then a random one will be selected and returned. These are for convenience, not complexity; if you want to avoid this behavior, just do a search on title or URL yourself and choose the entry you prefer.
The get methods can all be called directly from the class - you don't even have to create a WWW::KPR object to grab a username/password and use it to log into a given URL.
I also added a create_and_return that does what create is supposed to do: it creates an entry and returns the newly added UUID.
Finally, I tossed in a get_all that takes search parameters and returns an arrayref of hashrefs of all the matching entries.
See the POD for WWW::KeePassRest for the full documentation of the module's API, but in parting, I'll show you the much-streamlined version of the test script that exercises the module's API:
01 use strict; 02 use warnings; 03 use WWW::KeePassRest; 04 use Data::Dumper; 05 06 my $kpr = WWW::KeePassRest->new(); 07 08 # Search for keys with "Github" in the title 09 my $c = $kpr->get_all ('Github', 'SearchInTitles'); 10 print Dumper($c); 11 12 # Add a test key and get the UUID back. 13 my $newkey = $kpr->create_and_return ('test', { Title => 'wwwkpr test', 14 Password => 'test', 15 URL => 'http://test.cpan.org/wwwkpr' }); 16 print "$newkey\n"; 17 18 # Check it, modify it, delete it. 19 my $entry = $kpr->get ($newkey); 20 print Dumper($entry); 21 $entry->{'test-string'} = $newkey; 22 print $kpr->update ($newkey, $entry) . "\n"; 23 print $kpr->delete ($newkey) . "\n";
Cool, huh?
