Jan 15

If you were on /b/ today, you would have been redirected to the beta site. This was a part of a test conducted by 2ch engineers. The redirect is now off, but the numbers that came from our observation are (finally) optimistic.

We had tweaked a number of different settings and turned off different queries. When we had noticed a significant improvement of load times, /b/ was forwarded over. During that time, we peaked at over 60 requests per second without noticing performance problems. I will continue to work on getting Next operating with Varnish Cache, but it is still phenomenal to know that the application may run fast enough to handle the load without it. I've also been given some optimistic words regarding new hardware, but I'm not betting or planning on that happening as it remains out of my control.

After Varnish caching is introduced, the goal is then to run another import. I am shooting for the end of the month. The import process is going to be similar to last time. If you own a board, this is important information for you.

• As with last time, the site as a whole will remain up.
• There will be 4 importing processes instead of just one, each importing the (x*i+n)th board to speed things up.
• Boards being imported will be locked like last time.
• We will now copy images instead of symlinking them, which is going to alleviate a ton of attachment related problems we've had with the previous imports. This will come at the cost of speed.
• Attachments on our our largest and fastest boards will be skipped. I'm not sure what the threshold will be, but something like <50,000 posts sounds reasonable. Alternatively, I may decide to only import the first few (dozen? hundred? thousand?) attachments on each board to prevent 4 hour long import issues like we had with /aneki/ last time.
• YouTube embed links will be appended to the top of posts instead of discarding them.
• I will ensure existing posts's flags are imported because this is apparently a really big deal to the guy who runs /fringe/.

That's all I can think of. The Varnish layer is an extremely exciting prospect, but also very tricky. When I did tests, I was able to see 50 requests per second per core, which would be a gobsmacking increase over even the static content we're serving right now. That's the real goal here.

Josh