“It’s no longer one thing at a time, but everything all at once.”
Archive for July, 2011
Sorry to say, but our search engine is all kinds of weird this morning (San Francisco time). Lots of the pages you see around the site, like a Work page or a Subject page, or indeed the Search Results page are driven largely by search.
So, while we’re working on fixing the problem, please excuse the various gaps you might encounter around the place as we resolve it.
Update, 12:50pm PST: We’ve tried running Lucene CheckIndex on the work search index, but it came back with “No problems were detected with this index.” Hmm. Next, we’ll try restoring from backup. Apologies again for the continued oddness around the site!
Update, 2:30pm PST, 7/13/11: OK. We’ve just removed the site alert that you might have noticed across the top of every page, because we think the search is back up online. It turned out that we had to rebuild the whole index from a 2-day-old backup, and then process the last 2 days of changes made on the site into the new index. Phew! If you continue seeing weirdness or results that look less correct than usual, please let us know.
Apologies again for the road bump – and a word of advice? Be sure to keep an eye on how and where your log files are being stored, and accumulated. Turns out it was a huge blob of log files is what ended up choking our search engine, stopping it from being able to be updated. We’ve since modified where we store logs, and how often they are cleaned out.
I’ve been spending a bit more time on archive.org lately. Not only exploring the nearly 3 million scanned texts, but also our massive video collection. I uncovered this documentary showing earthlings landing something called “Phoenix” on the surface of Mars in June of 2008. Here’s what happened:
Go, humans! And, NASA!