Web key directory

[LecrisUT]

Can this expose the public keys according to GPG Web Key Directory?

It would also be helpful to guide on what DNS records need to be set and other such setups in order to get the keyserver to be auto-detected, and which clients/gpg versions support which setup.

[strausmann]

i just stumbled across this question, too.

[pravi]

There is a python script to generate .well-known/openpgpkey/hu/ from a keyring - https://hg.intevation.de/gnupg/wkd-tools/file/tip/generate-openpgpkey-hu, but we should probably use the existing database and map the wkd requests to hkps search requests.

[pravi]

The format that WKD expects,
https://intevation.de/.well-known/openpgpkey/hu/it5sewh54rxz33fwmr8u6dy4bbz8itz4 is the direct method URL for "bernhard.reiter@intevation.de".
and mailvelope keyserver's format is,
https://keys.puri.sm/pks/lookup?op=get&search=praveen%40debian.org (though this gives a html page and not raw key, so we will need to add another option to get raw keys).

It might be possible to do this url rewriting in web server as well. decode the request to email address and then rewrite the url to search by email address with an option to return raw data (probably in binary since wkd expects binary data - at least the example url returned binary data).

[pravi]

if we can get the hash generation logic figured out, then it can be simple I think.

1. generate the hash from email and store in the db when a key is uploaded
2. add a search option for this hash
3. do a url rewriting to map between wkd url format and mailvelope search url (in web server)
4. and add an option to return raw key in addition to current html page

Update: we can already pass options=mr https://keys.puri.sm/pks/lookup?op=get&search=praveen@puri.sm&options=mr

[pravi]

https://gist.github.com/kafene/0a6e259996862d35845784e6e5dbfc79 this is a useful resource. It shows how to generate the wkd-hash with gpg command itself. We need to find a way to do the same in Javascript.

[pravi]

https://github.com/openpgpjs/wkd-client/blob/master/src/wkd.js#L50 this has js code for generating the hash.

[pravi]

Based on wkd-client code, I was able to generate this hash using this code snippet.

const email = "bernhard.reiter@intevation.de"
    if (!email) {
      throw new Error('You must provide an email parameter!');
    }

    if (typeof email !== 'string' || !email.includes('@')) {
      throw new Error('Invalid e-mail address.');
    }

    const [localPart, domain] = email.split('@');
    const localPartEncoded = new TextEncoder().encode(localPart.toLowerCase());
    const localPartHashed = new Uint8Array(await crypto.subtle.digest('SHA-1', localPartEncoded));
    const localPartBase32 = encodeZBase32(localPartHashed);
    console.log(localPartBase32);
/**
 * Encode input buffer using Z-Base32 encoding.
 * See: https://tools.ietf.org/html/rfc6189#section-5.1.6
 *
 * @param {Uint8Array} data - The binary data to encode
 * @returns {String} Binary data encoded using Z-Base32.
 */
function encodeZBase32(data) {
  if (data.length === 0) {
    return "";
  }
  const ALPHABET = "ybndrfg8ejkmcpqxot1uwisza345h769";
  const SHIFT = 5;
  const MASK = 31;
  let buffer = data[0];
  let index = 1;
  let bitsLeft = 8;
  let result = '';
  while (bitsLeft > 0 || index < data.length) {
    if (bitsLeft < SHIFT) {
      if (index < data.length) {
        buffer <<= 8;
        buffer |= data[index++] & 0xff;
        bitsLeft += 8;
      } else {
        const pad = SHIFT - bitsLeft;
        buffer <<= pad;
        bitsLeft += pad;
      }
    }
    bitsLeft -= SHIFT;
    result += ALPHABET[MASK & (buffer >> bitsLeft)];
  }
  return result;
}

[pravi]

I manged to get the hash generated, but unable to pass the value back from util.js to public-key.js. Need to figure out doing this with proper async/await.

diff --git a/src/lib/util.js b/src/lib/util.js
index b871fba..271f0aa 100644
--- a/src/lib/util.js
+++ b/src/lib/util.js
@@ -87,6 +87,53 @@ exports.normalizeEmail = function(email) {
   return email;
 };
 
+exports.genWKDHash = function(email, wkdhash = '') {
+  const [localPart, domain] = email.split('@');
+  crypto.subtle.digest({
+          name: "SHA-1",
+      },
+      new TextEncoder().encode(localPart.toLowerCase())
+  )
+  .then(function(hash){
+     const localPartBase32 = encodeZBase32(new Uint8Array(hash));
+     wkdhash = localPartBase32 + '@' + domain;
+  })
+  .catch(function(err){
+      console.error(err);
+  });
+  console.log(wkdhash); 
+     return wkdhash;
+}
+
+function encodeZBase32(data) {
+  if (data.length === 0) {
+    return "";
+  }
+  const ALPHABET = "ybndrfg8ejkmcpqxot1uwisza345h769";
+  const SHIFT = 5;
+  const MASK = 31;
+  let buffer = data[0];
+  let index = 1;
+  let bitsLeft = 8;
+  let result = '';
+  while (bitsLeft > 0 || index < data.length) {
+    if (bitsLeft < SHIFT) {
+      if (index < data.length) {
+        buffer <<= 8;
+        buffer |= data[index++] & 0xff;
+        bitsLeft += 8;
+      } else {
+        const pad = SHIFT - bitsLeft;
+        buffer <<= pad;
+        bitsLeft += pad;
+      }
+    }
+    bitsLeft -= SHIFT;
+    result += ALPHABET[MASK & (buffer >> bitsLeft)];
+  }
+  return result;
+}
+
 /**
  * Generate a cryptographically secure random hex string. If no length is
  * provided a 32 char hex string will be generated by default.
diff --git a/src/modules/public-key.js b/src/modules/public-key.js
index bb0b2b1..0fe68f6 100644
--- a/src/modules/public-key.js
+++ b/src/modules/public-key.js
@@ -160,6 +160,10 @@ class PublicKey {
       if (userId.notify && userId.notify === true) {
         // generate nonce for verification
         userId.nonce = util.random();
+        // generate nonce for verification
+        console.log(util.genWKDHash(userId.email));
+        userId.wkdhash = util.genWKDHash(userId.email,userId.wkdhash);
+console.log("wkdhash" + userId.wkdhash);
         await this._email.send({template: tpl.verifyKey, userId, keyId, origin, publicKeyArmored: userId.publicKeyArmored, i18n});
       }
     }

[asdofindia]

@pravi

exports.genWKDHash = function(email, wkdhash = '') {
  const [localPart, domain] = email.split('@');
  crypto.subtle.digest({
          name: "SHA-1",
      },
      new TextEncoder().encode(localPart.toLowerCase())
  )
  .then(function(hash){
     const localPartBase32 = encodeZBase32(new Uint8Array(hash));
     wkdhash = localPartBase32 + '@' + domain;
  })
  .catch(function(err){
      console.error(err);
  });
     return wkdhash;
}

In the above, crypto.subtle.digest( function starts a new branch of execution (asynchronous) while the main branch continues from that line and jumps to return wkdhash;. At this point in time, wkdhash has not changed, and will be an empty string.

What needs to be done is to return the promise of crypto.subtle.digest itself, like this:

exports.genWKDHash = function(email, wkdhash = '') {
  const [localPart, domain] = email.split('@');
  return crypto.subtle.digest({
          name: "SHA-1",
      },
      new TextEncoder().encode(localPart.toLowerCase())
  )
  .then(function(hash){
     const localPartBase32 = encodeZBase32(new Uint8Array(hash));
     wkdhash = localPartBase32 + '@' + domain;
     return wkdhash;    // this is what the promise resolves to if there's no error
  })
  .catch(function(err){
      console.error(err);
  });
}

And then when you call it, you've to wait for the promise to resolve, like this:

userId.wkdhash = await util.genWKDHash(userId.email,userId.wkdhash);

---

This hashing operation seems to be destructive in that we cannot get the email address back from the hash. Therefore your latter comment about having to store the hash in the database seems right. Meaning for existing keys, there'll have to be a separate function to calculate the hashes and save to DB. And for new keys being uploaded, the hash can be calculated along with fingerprint, etc.

You could add a wkd endpoint like the hkp endpoint:

'use strict';

const Boom = require('@hapi/boom');
const util = require('../lib/util');
const openpgp = require('openpgp');

/**
 * An implementation of Web Key Directory
 * See https://wiki.gnupg.org/WKD
 */
class WKD {
  /**
   * Create an instance of the WKD server
   * @param  {Object} publicKey - an instance of the public key service
   */
  constructor(publicKey) {
    this._publicKey = publicKey;
  }

  /**
   * Public key lookup via http GET
   * @param {Object} request - hapi request object
   * @param {Object} h - hapi response toolkit
   */
  async lookup(request, h) {
    const hash = request.params.hash;
    // implement lookup
      return h.response(body).type('text/plain');
    }
  }
}

exports.plugin = {
  name: 'WKD',
  async register(server, options) {
    const wkd = new WKD(server.app.publicKey);

    const routeOptions = {
      bind: wkd,
      cors: options.server.cors === 'true',
      security: options.server.security === 'true',
      ext: {
        onPreResponse: {
          method({response}, h) {
            if (!response.isBoom) {
              return h.continue;
            }
            return h.response(response.message).code(response.output.statusCode).type('text/plain');
          }
        }
      }
    };

    server.route({
      method: 'GET',
      path: '/.well-known/openpgpkey/hu/{hash}',
      handler: wkd.lookup,
      options: routeOptions
    });
  }
};

But of course, implementing the lookup requires the hash to be saved.

[pravi]

@asdofindia thanks, let me try this.

Thanks also to Ananthu CV who also helped making this a promise.

diff --git a/src/lib/util.js b/src/lib/util.js
index b871fba..60d28eb 100644
--- a/src/lib/util.js
+++ b/src/lib/util.js
@@ -87,6 +87,47 @@ exports.normalizeEmail = function(email) {
   return email;
 };
 
+exports.genWKDHash = function(email, wkdhash = '') {
+ return new Promise((resolve, reject) => {
+    const [localPart, domain] = email.split('@');
+    const hash = crypto.createHash('sha1')
+      .update(localPart.toLowerCase())
+      .digest();
+    const localPartBase32 = encodeZBase32(hash);
+    const wkdhash = localPartBase32 + '@' + domain;
+    resolve(wkdhash);
+  });
+}
+
+function encodeZBase32(data) {
+  if (data.length === 0) {
+    return "";
+  }
+  const ALPHABET = "ybndrfg8ejkmcpqxot1uwisza345h769";
+  const SHIFT = 5;
+  const MASK = 31;
+  let buffer = data[0];
+  let index = 1;
+  let bitsLeft = 8;
+  let result = '';
+  while (bitsLeft > 0 || index < data.length) {
+    if (bitsLeft < SHIFT) {
+      if (index < data.length) {
+        buffer <<= 8;
+        buffer |= data[index++] & 0xff;
+        bitsLeft += 8;
+      } else {
+        const pad = SHIFT - bitsLeft;
+        buffer <<= pad;
+        bitsLeft += pad;
+      }
+    }
+    bitsLeft -= SHIFT;
+    result += ALPHABET[MASK & (buffer >> bitsLeft)];
+  }
+  return result;
+}
+
 /**
  * Generate a cryptographically secure random hex string. If no length is
  * provided a 32 char hex string will be generated by default.
diff --git a/src/modules/public-key.js b/src/modules/public-key.js
index bb0b2b1..bd04b5f 100644
--- a/src/modules/public-key.js
+++ b/src/modules/public-key.js
@@ -160,6 +160,8 @@ class PublicKey {
       if (userId.notify && userId.notify === true) {
         // generate nonce for verification
         userId.nonce = util.random();
+        // generate wkd hash
+        userId.wkdhash = await util.genWKDHash(userId.email,userId.wkdhash);
         await this._email.send({template: tpl.verifyKey, userId, keyId, origin, publicKeyArmored: userId.publicKeyArmored, i18n});
       }
     }

[pravi]

@asdofindia I tried adding src/route/wkd.js and visiting the .well-known/openpgpkey/hu/g4rf118d16sbj3k77d5txywpm879pszg which resulted in a 404. I think we should try to get a key not found/another debug message printed to get started before implementing the lookup part.

[pravi]

With the following code,

diff --git a/src/modules/public-key.js b/src/modules/public-key.js
index bb0b2b1..8fb8381 100644
--- a/src/modules/public-key.js
+++ b/src/modules/public-key.js
@@ -249,7 +252,7 @@ class PublicKey {
    * @param {string} keyId         (optional) The public key id
    * @return {Object}               The verified key document
    */
-  async getVerified({userIds, fingerprint, keyId}) {
+  async getVerified({userIds, fingerprint, keyId, wkd}) {
     let queries = [];
     // query by fingerprint
     if (fingerprint) {
@@ -265,8 +268,18 @@ class PublicKey {
         'userIds.verified': true
       });
     }
+    // query by wkd hash
+    if (userIds && wkd) {
+      queries = queries.concat(userIds.map(uid => ({
+        'userIds.wkdhash': {
+            $eq: uid.email},
+        'userIds.verified': {
+            $eq: true
+        }
+      })));
+    }
     // query by user id
-    if (userIds) {
+    if (userIds && !wkd) {
       queries = queries.concat(userIds.map(uid => ({
         'userIds.email': {
             $eq: util.normalizeEmail(uid.email)},
@@ -287,10 +300,10 @@ class PublicKey {
    * @param {Object} i18n          i18n object
    * @return {Object}               The public key document
    */
-  async get({fingerprint, keyId, email, i18n}) {
+  async get({fingerprint, keyId, email, wkd, i18n}) {
     // look for verified key
     const userIds = email ? [{email}] : undefined;
-    const key = await this.getVerified({keyId, fingerprint, userIds});
+    const key = await this.getVerified({keyId, fingerprint, userIds, wkd});
     if (!key) {
       throw Boom.notFound(i18n.__('key_not_found'));
     }
diff --git a/src/route/hkp.js b/src/route/hkp.js
index 7f90136..1604702 100644
--- a/src/route/hkp.js
+++ b/src/route/hkp.js
@@ -46,7 +46,7 @@ class HKP {
     const params = this.parseQueryString(request);
     const key = await this._publicKey.get({...params, i18n: request.i18n});
     if (params.op === 'get') {
-      if (params.mr) {
+      if (params.mr || params.wkd) {
         return h.response(key.publicKeyArmored)
         .header('Content-Type', 'application/pgp-keys; charset=utf-8')
         .header('Content-Disposition', 'attachment; filename=openpgp-key.asc');
@@ -84,7 +84,8 @@ class HKP {
   parseQueryString({query}) {
     const params = {
       op: query.op, // operation ... only 'get' is supported
-      mr: query.options === 'mr' // machine readable
+      mr: query.options === 'mr', // machine readable
+      wkd: query.options === 'wkd' // wkd hash
     };
     if (!['get', 'index', 'vindex'].includes(params.op)) {
       throw Boom.notImplemented('Method not implemented');

I'm able to get the raw ascii armored key when looking for /pks/lookup?op=get&search=g4rf118d16sbj3k77d5txywpm879pszg%40puri.sm&options=wkd

Now we just need to either add a wkd route or a url rewriting scheme in web server that will redirect to this url from the wkd direct url.

[pravi]

/pks/lookup?op=get&search=g4rf118d16sbj3k77d5txywpm879pszg%40puri.sm&options=wkd is what is working correctly now, what is expected is /.well-known/openpgpkey/hu/g4rf118d16sbj3k77d5txywpm879pszg

So we have two options to do this mapping,

1. Add a new route (previous try did not work)
2. Rewrite the url at web server

Domain name will need to dynamically selected based on the request url since the local part of hash is not unique.