22/04/2022
With a heavy heart we must announce to volunteers our intent to delay restarting the WCG until May 9th, 2022. Several issues discovered in our production environment remain unresolved, making it impossible to meet the April 22, 2022 deadline at this time.
Unexpected issues continue to delay the full validation of the QA environment, meaning there is no path yet to a responsible restart of the Production system even if all outstanding issues were resolved today. Several obstacles that proved difficult to resolve due to inexperience with specific components in the WCG software stack, contribute to our need for yet more time to bring the Grid back online.
Notably, the website build broke due to a dependency that brought React version 18.0.2 into the build, whereas the site was developed against React version 17. The lack of experience with React and modern web development practices within our team resulted in what now seems a simple fix, pinning React to the previously working version 17 for all dependencies that permitted a change in major versions when resolving required packages at build time. Volunteers may have noticed this issue, as it coincided with a long silence at www.worldcommunitygrid.org, which could not be updated. In addition, we overlooked a misconfiguration in the messaging/queueing middleware (IBM MQ), and a missing root certificate took far too long to discover as the reason Apache could not talk to IBM Websphere. The last few public IPs to be assigned were not routable due to a misconfigured VLAN. While these issues were all resolved, we now need more time to ensure there are no more surprises.
As a rule, this level of detail should be omitted from updates and has been omitted from previous updates. From our perspective, the specifics of the technical obstacles that hinder us are immaterial, as what we owe to volunteers is a working backend for the World Community Grid. The Grid is far too valuable to let go - and despite challenges, we are committed to supporting open science on the Grid. Given the already overlong timeframe of the migration, and to assuage concerns as to whether we are progressing towards the goal at all, we thought to make an exception given that we are asking for your patience just a while longer before we are fully ready to restart the WCG. We must and we will succeed.
Thank you to all who have contributed feedback and words of encouragement during the downtime. We do see your posts even if we cannot always reply at this stage. Your understanding and patience is truly appreciated. We will prepare a proper team introduction and answer the questions and address the comments once the Grid is back on.
WCG Tech team
