January 10, 10:48 Manila

A full technical write-up has been published: https://ghostbin.com/paste/vh627

Yesterday, someone offered to try setting up Infinity Next on a personal server. This server's hardware very closely matched what we have to use for 8chan; 8 cores, 8 gigs. It ran perfectly. Many of you may have experienced a brief redirect from /b/ or /v/ to an "odili.8ch.net" yesterday, this was us testing the hardware. This computer ran FreeBSD with PHP5.6-FPM (later PHP7-FPM), Postges, Nginx, and Redis.

A reformat of Five, our web server, was committed and now it runs Debian. This came after an expert network and system administrator from 2ch reviewed our stack. I've installed the older stable version of PHP5.6 and have tested it. The results are not good and it is increasingly frustrating to understand why.

This burngraph of the beta (http://i.imgur.com/EFpV0C6.png) shows a pretty slow response time of 450ms for the home page. Odili's server was easily dishing out similar replies in under 100ms, and Json replies especially were handled quickly. However, on our live node, it can take a full minute for a reply for no discernible reason. I, and more experienced web admins, cannot pinpoint the precise location of the problem.

I've said this before, but if this was a problem with just Next I would have no issue what so ever admitting personal failure. A similar issue happened in December where our Postgres driver was too slow. I had to take a few work days to write my own that was better than Laravel's. This is not the same vein of problem. For whatever fucking reason, there is a mystery lag before a response even begins to generate. It has absolutely nothing to do with load. At peak, odili's server was handling 40 requests a second at about 3~6 load out of 8. Next on 2ch's hardware is handling about 5 requests per second and burning out completely.

So at this point I am completely exhausted of options and am exhausted mentally. I've been working on this problem with over a week straight, it has been all I've done from waking up to going to bed. I have considered every possible configuration of settings and exploited every avenue of testing that I can think of. There is some great tragedy in this. If it had just worked right in December, we'd be wrapping up our first month post-release. I could have fixed all your bullshit gripes about JavaScript and CSS twice by now. Instead, I'm screwing around with ini and conf files for 14 hours a day. The most heartbreaking moment in recent history that I can think of is seeing Odili forward /v/ and having the fucking thing just work after all the effort I sank into trying to get the beta domain up and running. It was surreal.

With copypaste/hotwheel's permission, I'm surrendering the network details over to this rando from -dev so he can look at it. He's already discovered that 2ch's hardware is actually only 4 physical cores and 8 logical cores, so who knows what else he can turn up. Maybe we're running on the shittiest server out of all of them or something. I don't know. I can't think of anything anymore.

~ Josh