We're drowing in email again. We're working on purging a bunch of dead emails, spams, bounce messages, etc., and have the Postfix mail server daemon shut down until we can get the queues purged. Please bear with us.
We are hoping to begin the long prophesied moved to newer server hardware in the next few weeks. When that happens, we'll have at least a couple of periods of several hours' system unavailability as we move userids, files, mail spools and such to the new server. Further updates once the dates are nailed down.
Would everyone please go through their email inboxes and purge any messages no longer needed? If you wish to retain messages, please create a "mail" directory in your home directory and move messages there. This will help reduce usage on the incoming mail directory, which is short on space.
We're currently purging several thousand spam emails generated against a compromised userid from the mail queue. To expedite the process, we have the mail system sending agent (Postfix) shut down. Once the purge jobs are finished running we will bring Postfix back up. At the rate things are going, we should be done by around 4:00pm.
We are evaluating the current state of the AFN email service, with an eye to getting us off the blacklists (e.g. Yahoo and Hotmail are currently rejecting afn.org email), enhancing protection against incoming spam, and improving the performance of the mail server. As we work with the server configuration there may be periodic interruptions in email access and delivery. We will work to minimize any such disruptions, and apologize for any inconvenience caused by this work.
We have disabled SSLv2 and SSLv3 on incoming and outgoing email and web services, as both protocols are vulnerable to various attacks. STARTTLS will still work.
New domain name ssh.afn.org has been added to replace ftp.afn.org, since the FTP protocol has been discontinued. Using ssh.afn.org is preferred to explicitly naming freenet1.afn.org or other freenet machine, as it will be kept current regardless of which machine is hosting the SSH service.
Apologies for the server move running over the end of the maintenance window. When we moved the server farm to its new rack, we discovered that the power in the rack was 208V instead of 120V. We also discovered that our Promise Ultratrak RM8000 RAID array did NOT have autoswitching power supplies. There were loud bangs, flashes of light, and small clouds of acrid smoke.
Fortunately, the University of Florida's Astronomy department had a surplus Ultratrak RM8000, which they were very gracious to loan us (thanks, Matt!). After moving the drives to the new array housing and TRIPLE checking that the power supplies were set to 230V, we were able to bring the system back up at about 4:20pm.
The AFN DNS nameserver will move to a different machine in the AFN cluster. The change should not be service affecting.
The AFN server farm will be relocated to a different rack in the UF SSRB Data Center. All AFN services will be unavailable during the server moves. We do not anticipate the moves taking more than an hour, but maintenance window is scheduled for two hours in case we run into any snags. We have to vacate our current rack and reduce our in-room footprint to make room for new Data Center equipment.
Due to spammers finding and sending to email@example.com,
the address has been changed. Send requests for assistance to "request" at afn.org.
Most systems are working now. Unfortunately, the current system as rebuilt will require
everyone to have their account password reset. It can be reset to the previously used password,
but the new system uses a different password encryption method, making the updates necessary.
HOWEVER, SOME USERS HAVE REPORTED SUCCESSFUL CONNECTIONS WITHOUT HAVING TO UPDATE THEIR PASSWORD.
Please try to connect as usual before requesting a password update.
The authentication process after log on may take up to 10 seconds. To have your password reset,
send an email message (or have a friend send for you) to
firstname.lastname@example.org (see news above for
April 21st) with your name and a phone number where you can be reached; someone will text or call
that number within a few days to coordinate the password reset. Let us know in your email
if you can't receive text messages.
Several changes have been made to enhance security and performance. Telnet and FTP are not available, as those older services pass usernames and passwords in clear text, unencrypted. Instead, SSH replaces telnet and SFTP/SCP replaces FTP. There are several applications available for this, such as PuTTY for SSH, with brief instructions, and WinSCP for SCP for Windows; Mac Terminal utility with SSH and Fugu for Mac OS X. Note that fugu-1.2.1pre1 in the Unstable folder is the only option now available for Mavericks. Use ssh.afn.org as the domain name to connect to either service.
IMAP for email uses TLS security, and clients must be configured with TLS for connecting to imap.afn.org for receiving and smtp.afn.org for sending. POP email protocol is no longer available.
We're currently working on getting authentication working again for email. The IMAP server will come and go through the weekend until we get the server and the LDAP directory happy with each other. We're not there yet, but we're getting closer!
Good news! Looks like we've got the web server back! Users should be able to see web pages originating from their public_html directories. Email is still not there, but we're getting closer. Stay tuned!
New servers have been installed and configuration is ongoing. As soon as we can retrieve some configuration files from one of the dead nodes, we ought to have email and web services restored, hopefully within the next day or two.
Once everything is back up and stable, we'll let things be for a few weeks, then schedule a maintenance window to migrate user services and data off the interim server onto the permanent replacement server.
AFN has suffered a series of hardware hassles over the last several days, culminating in several tired servers more or less simultaneously giving up the ghost. This has lent a certain previously lacking urgency to our long standing plans to migrate off off the hodge-podge of elderly scrounged and donated servers.
AFN sysadmins are now engaged in first phase of a multi-step process to move to newer hardware and more recent versions of server software. The interactive, email/fileserver/web, and DNS services are being moved this weekend. Assuming the installation and configruation tasks go smoothly, and the interactive node's user and password files can be recovered relatively painlessly from the dead drives, we should have interactive, email and web services back up by the first part of the week of March 10th.
The next step in the process will be to move the users' home directories onto a much newer, more capable, more reliable fileserver. Once the home directories have migrated, the last phase of the project, migrating services onto the big server will take place. At that juncture, AFN should be on a much better hardware and software footing going forward.
We do apologize for the disruption in service caused by the raft of system issues and their mitigation. We expect the situation to be much improved after we reach the end of the migration project.