wget: “Issued certificate has expired” after September 30, 2021

Two websites that I download data from using automated processes stopped giving me new data from October 1. When I investigated the problem, I could see an error message from the wget program in Linux:

Connecting to SOME.HOSTNAME (SOME.HOSTNAME)|1.2.3.4|:443… connected.
ERROR: cannot verify SOME.HOSTNAME’s certificate, issued by ‘/C=US/O=Let’s Encrypt/CN=R3’:
Issued certificate has expired.
To connect to SOME.HOSTNAME insecurely, use `–no-check-certificate’.

The quick fix, obviously, was to add the –no-check-certificat to the command line, which allows the download to go ahead, but what’s the root cause? My assumption was that the site owner had let an SSL certificate expire, but after it happened with a second site from the same date, I got suspicious. It turns out, Let’s Encrypt which is used by many websites for free encryption certificates previously had a certificate that expired on September 30 and which has been replaced by a new certificate but many pieces of software don’t retrieve the new certificate. That’s because it’s signed with a new root certificate that a lot of older software don’t trust yet. They need an updated of the root certificate store.
In my case, running

sudo yum update

would update the ca-certificates package and that allowed wget to trust the new certificate.
Please see these links for more information:

Rescuing DVD-RAM Recordings from Obsolescence

I bought my first video camera more than 30 years ago. I went for Hi8, a higher resolution version of Video8, as opposed to VHS or VHS-C which was also popular at the time. Since then I have switched device and formats several times. There was always the worry that I would lose access to my recordings as devices able to read the old media become obsolete or die or the recording media themselves fail from old age. Losing irreplaceable videos of our kids when they were little is something I really didn’t want to experience. Here is a short summary of how I have been dealing with these challenges.

Hi8 (PAL) – early 1990s
I bough my first camcorder when I still lived in Germany so naturally it used the PAL standard (625 scan lines, 50 Hz). I did a lot of analog video editing using an S-VHS VCR, which could interface to the camcorder using S-Video cables. Even after I moved to Japan which uses the NTSC standard (525 scan lines, 60 Hz) I kept recording in PAL. A Samsung multi-standard recorder allowed me to record from the camcorder to NTSC VHS tapes. I also bought a multi-standard analog TV that could display PAL, SECAM and NTSC. However, for many years I just collected the Hi8 PAL master tapes in a cardboard box.
Along came Digital8, a successor to Hi8 that as the name indicates used digital recording but was backwards compatible with Hi8 and could play the old tapes. So eventually, as Hi8 camcorders were already becoming obsolete, I bought a second hand one off eBay when I was visiting Germany. It had an IEEE 1394 (Fireware) connector that made it possible to copy digital video to a computer equipped with that interface. I experimented with PCs with plug-in IEEE 1394 cards, but ultimately it was a Mac mini that allowed me to copy the old Hi8 PAL tapes to a hard disc using the German Digital8 camcorder, a Firewire cable and iMovie which was bundled with macOS. The output files were “.dv” files. Some tapes were difficult to load and took many tries before the camcorder would even play them, but I was largely successful.

Hi8 (NTSC) – late 1990s
When my kids started going to kindergarten I finally switched to a Japanese camcorder, still a Hi8 model but for the NTSC standard (US/Japan). Like with the PAL camcorder I saved all the tapes in a box. The Samsung multi-standard VCR developed issues and we bought a new S-VHS VCR equipped with a DVD drive. It supported DVD-R, DVD-RW and DVD-RAM, the latter with caddies (cartridges). It also supported S-Video. It was relatively easy to use the S-Video interface to copy from the Hi8 (NTSC) camcorder to double sided DVD-RAM media. About 2 hours of video would become one 4.2 GB file on DVD-RAM. I chose DVD RAM because it supposedly was more robust than DVD-RW (especially with the protective case), but as Blu-ray came along DVD-RAM became less and less common, with many DVD multi drives not supporting it any more. In 2008 and 2010 I made a stack of 6 double sided DVD RAM media that held video from 21 Hi8 NTSC tapes, but when the DVD section of the VCR died, I no longer had anywhere to play them.
This year I finally bought a USB 3.1 DVD drive that also supports DVD-RAM, though without support for caddies. I went for the BUFFALO DVSM-PTV8U3-BK/N (2180 yen, about US$21). It worked very well once I removed the DVD-RAM disks from their protective caddy. I hooked it up to a Windows 10 machine, plugging the two USB cables into different USB ports (USB 2.0 for power, USB 3.1 for data). I copied the entire folder structure on each side of the media to a separate new folder on the server hard disk. The actual video information on a DVD RAM disk is in a file called VR_MOVIE.VRO which is found inside a folder called DVD_RTAV. The open source VLC player will play this .vro file, as well as the .dv file captured on the Mac mini from the Hi8 recorder.

MPEG (NTSC) hard disc recorder – 2000s
After the various Hi8 recorders I moved from tape to a hard disk based camcorder, a Toshiba Gigashot GSC-R30. This used a Toshiba-made 1.8″ notebook hard disk. It also had a USB 2.0 interface and could be connected to any PC. The MPEG files would play on any MPEG player with support for its audio codec, including VLC. Therefore backing up and preserving these videos was pretty painless.

Smartphones – 2010s and beyond
The Toshiba GSC-R30 was the last camcorder I ever bought. Occasionally I still shot video on my Nikon D3300 DSLR camera, but mostly I moved on to mobile phones which may not have had an optical zoom or as much recording capacity, but they were always in your pocket and so easy to use and the quality improved with each generation.

Don’t lose your media!
If you value the images and videos you recorded over the years, make sure to migrate them to recording media that you can access for years to come and keep doing that. Also make sure you have backups. Tapes, DVDs, hard disks and SSDs will all become unreadable at some point. Don’t keep irreplaceable files on one laptop and hope that it will work forever because it won’t. At the very least, buy a USB drive and make a backup copy. Even better, buy another USB drive, make another backup copy and give it to someone else in your family. It’s better to have your valuable data saved in more than one place.
I am a great fan of the VideoLAN VLC media player. You can throw just about any video or audio format at it and it will be able to play it. I highly recommend it! 🙂

Outlook Express Error 0x800CCC0B and the End of TLS 1.0 (Deprecated SSL Protocol)

Microsoft Outlook Express (OE) is an obsolete mail client that was available in Microsoft Windows XP, Windows 2003 Server and older Microsoft operating systems. It was no longer available on Windows Vista and later, though Windows Live Mail is relatively close in user interface and appearance.

Despite being obsolete and only working on operating systems no longer supported or updated by Microsoft, it still has some users who prefer its simple but powerful user interface. Some of those users will have had a frustrating experience recently, when various mail servers stopped working for outbound mail in OE. Specifically, these are mail servers that use SSL on submission port 465 or 587 for SMTP.

Secure Socket Layer (SSL) is a mechanism for encrypting data between a client and a server. You may know it from website URIs starting with “https:” and web sessions displaying a padlock symbol next to the URI. There are various protocol versions that can implement this encryption layer. One of these, TLS 1.0 which was conceived in 1999, has now been officially deprecated (made officially obsolete) as of the end of June 2018. Software now has to use more recent protocols, such as TLS 1.1, TLS 1.2 or the recently defined TLS 1.3.

Unfortunately, TLS 1.0 is all that OE will speak. It does not understand TLS 1.1 or later. Therefore it can not pick up mail from a POP server using SSL on port 995 or an IMAP server on port 993 or send mail to an SMTP server on port 465 (or 587) with SSL enabled.

Workaround
The only workaround I am aware of (other than switching to a more modern mail client) is to use Stunnel, a tool for Windows or Linux that acts as a proxy. You can configure it to establish an SSL connection to a given host and port when a connection to a given local port is made. Thus you could configure OE to connect to port 9465 on the machine running Stunnel, which might then connect via SSL to smtp.example.com:465 using a more modern TLS version supported by Stunnel (but not directly by OE).

Example
Let’s say Outlook Express was configured to submit outbound mail to smtp.outboundmailserver.com, port 587 via SSL/TLS. This is our SMTP server. Once this server refuses to allow TLS 1.0 connections, Outlook Express will no longer work. Let’s say we also have a simple Linux server mylinuxserver.com. This could even be something like a Raspberry Pi single board computer booting off flash memory. It can run on a local IP in our LAN, if you don’t need to have access from outside your building (OE running on a desktop). On this server we install the stunnel package:

sudo yum install stunnel

Please read the documentation on how to enable the service and have it auto-start when the Linux server reboots.

Next we configure stunnel to act as a client on our behalf and configure it to accept TLS 1.0 connections from us and forward them to the real POP3, SMTP or IMAP server using the latest TLS on our behalf. We will create lines like these in /etc/stunnel/stunnel.conf:

client = yes

;cert = /etc/pki/tls/certs/stunnel.pem
;sslVersion = TLSv1
;chroot = /var/run/stunnel
;setuid = nobody
;setgid = nobody
;pid = /stunnel.pid
;socket = l:TCP_NODELAY=1
;socket = r:TCP_NODELAY=1

[smtp-outboundmailserver]
accept = 1587
connect = smtp.outboundmailserver.com:587

Create other entries for the services that you need TLS support for and restart the stunnel service. Then reconfigure Outlook Express to access the Linux host and the port number listed with “accept = ” in place of the original server that refused your Outlook Express TLS 1.0 connection. You should be good to go!

Long term you will still need to migrate to another mail client such as Thunderbird, Windows Mail or OE Classic, but this workaround will buy you some time for that.

Picasa: “Failed to download album list”

If you are still using the Picasa 3 desktop application by Google and got the above error message, here’s some bad news for you: Google has finally killed this app. On March 26, 2018 they announced that it would no longer be able to upload new albums. So this error message is not temporary and there is no direct fix.

I think it’s very regrettable that Google has been killing off Picasa step-by-step. This is only the latest nail in the coffin. I had been using Picasaweb and Picasa since 2010 and they were great products.

The good news is that you can still create albums from folders using a web browser. Say you have a folder named “2018-03-26 Cherry Blossom Party”. Just follow these steps (for Windows and Chrome):

1) Select its parent folder in Windows Explorer, then slowly click on the folder that you want to upload, twice: Once to select it, then once more to enable you to edit the folder name as if to rename it. When the name becomes editable, press Ctrl+C to copy the folder name, then press Esc to keep the name unchanged. This stores the folder name in the copy-and-paste clipboard, which will save you from having to manually retype the name later.

2) In Chrome, go to https://photos.google.com/ and click on “Upload” (on the top right). A file selector dialog will open up. Click through to the contents of the folder you want to upload. Select all files in the folder using Ctrl+A and click “Open” to confirm the upload.

3) The browser will upload all files and give you a choice of “Add to album” or “Shared album”. Select “Add to album”. To create a new album with the name of the folder, select “New album”. Click on the album name showing as “Untitled” and use Ctrl+V to paste the name you copied in step 1. Hit Enter and click on the check mark to confirm creation of the new album.

Voila, you have a new album online, with the same name as the local folder. Repeat as needed for multiple folders. This is as simple as it gets without the old Picasa app.

Adding Free SSL Certificates for HTTPS To Your Websites

I recently received a warning email from Google:

“Starting October 2017, Chrome (version 62) will show a ‘NOT SECURE’ warning when users enter text in a form on an HTTP page, and for all HTTP pages in Incognito mode.

The recommended solution was to migrate the affected website(s) to HTTPS. This requires an SSL certificate. There are many companies selling those for hundreds of dollars. I didn’t really want to spend that money.

It turns out there is a free alternative: The Let’s Encrypt project (https://letsencrypt.org/) provides free SSL certificates with just enough functionality to run SSL with current browsers. It also provides automated tools that greatly assist you in obtaining and installing those certificates.

I had a default SSL host configured on my Apache 2.4 installation (inherited from a different server running Ubuntu) that I had to manually remove.

Then, when all virtual hosts only had port 80 (HTTP) enabled, I could run the certbot tool as root:

# certbot --apache

It enumerates all host names supported by your Apache installation. I ran it repeatedly, for each domain and the corresponding www. host name (e.g. joewein.net, www.joewein.net) in my installation and verified the results, one at a time. It will create a new virtual host file in /etc/httpd/hosts-enabled for those hosts for port 443 (HTTPS). I appended the content of that file to my existing port 80 (HTTP) virtual host file in /etc/httpd/hosts-available for that host name and deleted the new file created by certbot. That way I can track all configuration details for each website for both HTTP and HTTPS in a single file, but this purely a personal choice.

All it takes is an Apache restart to enable the new configuration.

You can test if SSL is working as expected by accessing the website with a browser using https:// instead of http:// at the start of the URI.

If you have iptables rules for port 80, you may want to replicate those for port 443 or the certificate generation / renewal may fail. Also, you want to make sure that SSLv3 is turned off on your Apache installation, to protect against the POODLE vulnerability. This required the following setting in ssl.conf:

/etc/httpd/conf.d/ssl.conf:SSLProtocol all -SSLv2 -SSLv3

The free certificates will expire in 90 days, but it’s recommended to add a daily cron job that requests renewals so that an updated key will be downloaded after 60 days, long before the old key expires. Once that is in place, maintenance of SSL keys will be totally automatic.

UPDATE (2017-11-01): If you’re using WordPress on your website, you should change the WordPress base URI to HTTPS too. To do that, log into the WordPress Dashboard. In there select Settings > General. Change the “http://” in the WordPress Address (URI) and Site Address (URI) fields to “https://” and click the Save Changes button. This ensures that any messages from WordPress to you will include secure URIs.

Porting iptables to ip6tables

A couple of days ago I received an email notification by the Berkeley Security Notifications Team that a Linux server of mine had less restrictive firewall rules for IPv6 than it had for IPv4. This prompted me to update my ip6tables settings on that host to make it is as secure via IPv6 as it was for IPv4.

If you have a dual stack server with IPv4 A records and IPv6 AAAA records published in DNS, you should have it protected with firewall rules on both protocols. Even if you only publish A records and not AAAA ones, you should secure IPv6 access because its address may leak to potential attackers in other ways.

The ip6tables tool is installed as part of iptables on recent distributions, but you need to set up one set of rules for each protocol. They’re independent of each other. A (not very secure) default ip6tables configuration might look like this:

# Generated by ip6tables-save v1.4.21 on Thu Sep 24 11:17:56 2015
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1456:118498]
-A INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 22 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT –reject-with icmp6-adm-prohibited
COMMIT
# Completed on Thu Sep 24 11:17:56 2015

It’s relatively easy to port additional settings from iptables to ip6tables (e.g. in /etc/sysconfig/iptables and /etc/sysconfig/ip6tables for CentOS).

Below are some of the changes needed when porting common entries. As you can see, some names are replaced with those of IPv6 equivalents. Any IP addresses and CIDRs for ip6tables need to be written in IPv6 notation.

To easily port over IPv4 addresses, simply prefix them with “::ffff:”. If they’re followed by a bit count such as /24 (the routing prefix size), add 96 to that number (IPv6 addresses are 128 bits each versus 32 bits for IPv4). Add equivalent rules for the corresponding native IPv6 addresses as needed.

  1. Accept ping from any source:

    IPv4:

    -A INPUT -p icmp -j ACCEPT

    IPv6:

    -A INPUT -p ipv6-icmp -j ACCEPT

  2. Accept connection from white-listed address:

    IPv4:

    -A SSH-IN -s 123.45.67.89/32 -j ACCEPT

    IPv6:

    -A SSH-IN -s ::ffff:123.45.67.89/128 -j ACCEPT
    -A SSH-IN -s 2345:abcd:678:42::/64 -j ACCEPT

  3. Rule to block access (after all the exceptions):

    IPv4:

    -A INPUT -j REJECT –reject-with icmp-host-prohibited
    -A FORWARD -j REJECT –reject-with icmp-host-prohibited

    IPV6:

    -A INPUT -j REJECT –reject-with icmp6-adm-prohibited
    -A FORWARD -j REJECT –reject-with icmp6-adm-prohibited

Filco Majestouch-2 [FKBN104M/EB2]

Recently, the space bar of the keyboard on my main machine developed a problem, so I ordered a Filco Majestouch-2 (US layout, USB version with PS/2 adapter). It uses brown Cherry MX switches.

I have always liked the feel and feedback of the original IBM PC and IBM PC/AT keyboards (which I first used in 1981). If you’re a fan of the original IBM keyboards, you’ll love this one. The Filco keyboards are not cheap, but you get what you pay for.

There are various models from Filco, some with the blue or black Cherry switches. The brown switches are recommended for general use, including office work and programming. I am very happy with mine and will probably order another one for another of my machines.

Acer One D260 system restore

The hard disk in my wife’s Acer One D260 netbook got damaged. A new hard disk is about a quarter the price of a new netbook, so I wanted to install a new drive. Like with most PCs these days there aren’t any Windows install DVDs included.

The netbook came with Windows 7 Starter, which we needed to somehow install on the new hard disk. Fortunately, the damaged hard disk was still limping along enough to use the Acer eRecovery system to create two Recovery DVDs. These should allow restoring the initial system state to a hard disk in the machine, wiping all the data on the drive.

To replace the hard disk, I had to undo seven clips around the edge of the keyboard, lift off the keyboard and disconnect the keyboard ribbon cable to the motherboard connector. Then I needed to undo 4 screws underneath and push through, to pop out the cover on the bottom of the machine. This opened access to the single memory slot and drive cage.

The 1 GB memory module on the motherboard can be replaced with a 2 GB PC3-8500 1066MHZ DDR3 module available for about $20. This is a wortwhile investment and I already have the module on order.

I replaced the damaged 250 GB WD Scorpio Blue drive with a spare 500 GB drive (available new for about $60-$80). Then I closed the cover and reinstalled the screws and then the keyboard.

With the new drive it was possible to boot off the first Recovery DVD using a USB DVD drive. The eRecovery software copied data from both DVDs to the hard disks and then rebooted. However, that reboot failed because the new drive did not yet have a Windows Master Boot record (MBR) on it. You can install an MBR from within Windows, but not from the bootable eRecovery DVD. So I had a chicken and egg problem.

I overcame this hurdle by booting off a Ubuntu Live DVD (32 bit), installing the ‘lilo’ package and telling it to install the Linux equivalent of Microsoft’s MBR code:

sudo apt-get install lilo
sudo lilo -M /dev/sda mbr

At the next attempt to boot off the hard disk, Windows started installing its components and drivers and launched into its initial configuration, just like the first time we had unboxed the machine more than two years ago. So we are back to a working Winmdows 7 machine!

Thank you, Linux — you saved my day again! 🙂

Western Digital 4 KB sector drive alignment for Windows XP and 2003 server

If your existing Windows XP or Windows 2003 Server machine needs a new C: drive, there are ways of upgrading to one of the latest drives without a complete software reinstall, but you may encounter some stumbling blocks due to the new Advanced Format technology, which uses 4 KB sectors.

When one of my PCs developed hard disk problems and I had to upgrade one of its drives, I also checked out my other machines. I found the C: drive of a Windows 2003 Server machine was about to fail. Windows 2003 is basically the server version of Windows XP, with which it shares most components. I opted for a 1 TB WD Red drive (WD10EFRX) by Western Digital, since these drives are designed for 24/7 operation, primarily for use in Network Attached Storage (NAS) appliances (desktop drives are only designed for an 8 hours on, 16 hours off use pattern).

I did not want to reinstall everything from scratch on that machine, so I used a Linux boot DVD and the GNU dd utility to mirror the failing drive onto the new WD Red drive (“sudo dd if=/dev/sda of=/dev/sdb”). As a result, all the partitions were in the same place and the same size as on the old drive, a Seagate Barracuda 7200.11 320 GB. The partitions on the old drive had not been aligned on 4 KB boundaries as is recommended to get decent performance on modern Advanced Format drives, so I needed to run an align tool to move the partition to the proper place. Western Digital offers one free to its customers, so that should be easy then, right?

No quite. I encountered all the troubles described by others in this thread: Basically, the download link for the WD Align tool (AcronisAlignTool_s_e_2_0_111.exe) takes you back to the same page, over and over, without error message. It turns out that you need to be registered and logged in to the WD site for the download link to do anything. You need to register both your contact details (name, e-mail address, postal address, phone number) and your hard disk’s serial number. For the latter I had to shut down the machine again and take out the drive once more to take a look, because the number is not printed on the cardboard box, only on the drive itself.

Once I registered my new drive, a download link did appear next to the registered product, but from it I found I could only download Acronis True Image and not the Acronis Align Tool (Advanced Format Software, WD Align). The WD Red series drives are all Advanced Format Drives, as is pretty much every drive made since 2011, but WD say it is designed for NAS use and hence don’t see the need for a fix for what they see as a Windows XP problem.

Various people online recommended a download site in Ukraine that apparently offers a copy of that program, but if you’re downloading from sites like that you risk installing malware on your computer. Beware!

There is a safer solution. I had to register another Western Digital drive, an old WD10EARS to get a usable download link for Advanced Format Software. If you don’t happen to have one lying around, a Google image search for WD10EARS will show you many photographs of disk drives with clearly readable serial numbers on the label. And apparently, these serial numbers will do the trick! 😉

After I downloaded the software, I ran it to make a bootable CD (it also seems to be Linux-based), booted and ran it and 1 hour and 30 minutes later my C: partition was showing up as properly aligned.

I can understand that Western Digital wants to restrict the use of licensed Acronis software to its own customers, denying other brands a free ride. However, the hoops it is making people jump through to be able to use one of their new drives as an upgrade to an existing Windows XP machine is just ridiculous. If a login is required to do the download, it should clearly say so. And if a drive uses 4 KB sectors (Advanced Format), its serial number should qualify you for the download. There are millions of existing XP users out there still and many will need new hard disks before they need a new computer.

Upgrading to a Western Digital WD20EFRX hard disk

All hard disks will die, sooner or later. They only way to avoid that is to retire a drive early enough. Often I upgrade drives because I run out of disk space, and migrate the data to a bigger drive. However, this times it looks like one of my drives is about to die.

Over the last couple of months, one of my PCs that is processing data 24/7 has been seizing up periodically, so I was starting to get suspicious about its hard drives (it has two of them). This week the Windows 7 event viewer reported that NTFS had encountered write errors on the secondary drive. It’s a Samsung SpinPoint F2 EG (Samsung HD154UI, 1.5 TB) which basically has been busy non stop for over three years.

I installed smartmontools for Windows and it showed errors:

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 099 065 051 Pre-fail Always - 5230
(...)
13 Read_Soft_Error_Rate 0x000e 099 065 000 Old_age Always - 5223
(...)
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 12379
(...)
197 Current_Pending_Sector 0x0012 099 099 000 Old_age Always - 24

“Reported_Uncorrect” are fatal errors and “Current_Pending_Sector” are bad sectors the drive wants to replace with spare sectors as soon as it can. Neither is a good sign. So I have ordered a new drive, started a backup to another machine and will replace the drive with a new disk that I have ordered from Amazon.

The new drive is a 2 TB Western Digital WD20EFRX, which is part of WD’s “Red” series. These drives are specifically designed for 24/7 operation (as opposed for 8/5 office computers). The drive is 0.5 GB bigger, which is just as well as the old drive was getting close to filling up. Gradually I will be moving my processing to an Ubuntu server, which I already use as my main archive machine with a RAID6 drive array.