Openssl por defecto lo genera en PEM y tu tienes que transformalo en DER
Te recomiendo usar algo más "fuerte":
openssl genrsa -aes256 -out <name of your certificate>.key 4096
des 1024 es muy flojo
Para crear el certificado ssl (https) del foro sólo se necesita crear el .key y el .csr, no he usado nunca el .pem ni el der.
PEM
Can contain all of private keys (RSA and DSA), public keys (RSA and DSA) and (x509) certificates. It is the default format for OpenSSL. It stores data Base64 encoded DER format, surrounded by ascii headers, so is suitable for text mode transfers between systems.
DER
Can contain all of private keys, public keys and certificates. It stored according to the ASN1 DER format. It is headerless - PEM is text header wrapped DER. It is the default format for most browsers
x509 certificates and RSA keys can be stored using a number of different formats. Two common formats are DER (a binary format used primarily by Java and Macintosh platforms) and PEM (a base64 representation of DER with header and footer information which is used primarily by UNIX and Linux platforms). There is also an obsolete NET (Netscape server) format which was used by earlier versions of IIS (up to and including 4.0) and various other less common formats which are not covered in this article.
Convert a PEM file to DER
openssl x509 -outform der -in certificate.pem -out certificate.der
Aqui explican algo más de la conversión:
http://gagravarr.org/writing/openssl-certs/general.shtmlhttp://support.citrix.com/article/CTX106631/