tag:blogger.com,1999:blog-14455177.post7533186217653580437..comments2024-03-07T18:57:25.977+01:00Comments on Mikael Ronstrom: Restart phases of a node restart in MySQL ClusterMikael Ronstromhttp://www.blogger.com/profile/07134215866292829917noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-14455177.post-76046543554102947772015-03-21T00:01:31.519+01:002015-03-21T00:01:31.519+01:00Restarting more than 1 node at a time freaks me ou...Restarting more than 1 node at a time freaks me out a bit... I know it's strictly psychological, but still... hahdigitalpointhttps://www.blogger.com/profile/13595987087265745150noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-73016290737877249682015-03-20T09:52:34.424+01:002015-03-20T09:52:34.424+01:00The comment about MaxDiskWriteSpeed requires a len...The comment about MaxDiskWriteSpeed requires a lengthy reply which I will respond to in a new blog rather. I was actually planning to write this blog anyways. So will arrive a new blog on the topic later today.<br /><br />If you do an upgrade you can actually also upgrade more than 1 node at a time. Many phases (but not all) are running in parallel. So oftentimes people run upgrades of 8 nodes by restarting 4 nodes (1 node per node group) simultaneously. So this means that you can probably slice your upgrade time even down to 1 hour.Mikael Ronstromhttps://www.blogger.com/profile/07134215866292829917noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-13355000083680009672015-03-20T04:53:55.356+01:002015-03-20T04:53:55.356+01:00Just a followup... did a rolling restart today for...Just a followup... did a rolling restart today for the first time with everything already on 7.4. 60GB nodes restarting in about 20 minutes now. A massive improvement to be able to do all 8 nodes in 2.5 hours, vs. closer to 8 hours total under 7.3.<br /><br />Now I just have to figure out why the data nodes are using so much disk i/o. No matter what's going on, they will use 100% of MaxDiskWriteSpeed. Maybe I'm just not understanding what it's intended for, but seems strange that if you set it to 400MB, ndbd uses 400MB constantly... set it to 20MB stuff still runs fine, but it only does 20MB disk i/o. What is it doing that it has to saturate whatever disk i/o is allowed? loldigitalpointhttps://www.blogger.com/profile/13595987087265745150noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-80373033916641864352015-03-07T02:21:44.673+01:002015-03-07T02:21:44.673+01:00Good catch, they can definitely be changed as part...Good catch, they can definitely be changed as part of node restart, so no need to shut down the cluster for this, I will post a bug report about the docs for this.<br /><br />Also for the brave :) I added some undocumented DUMP codes to update those even in a live node<br />as part of the 7.4 development.<br /><br />Read in Backup::execDUMP_STATE_ORD on the how to. These codes are mostly intended as a last resort, we haven't developed a proper interface yet to changing these kind of things online. Can only be used against 7.4 nodes, will be ignored by 7.3 nodes.<br />Mikael Ronstromhttps://www.blogger.com/profile/07134215866292829917noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-22413860817932438792015-03-06T23:39:33.742+01:002015-03-06T23:39:33.742+01:00Just noticed the new parameters require a full clu...Just noticed the new parameters require a full cluster shutdown/restart...<br /><br />MinDiskWriteSpeed<br />MaxDiskWriteSpeed<br />MaxDiskWriteSpeedOtherNodeRestart<br />MaxDiskWriteSpeedOwnRestart<br /><br />"System restart: The cluster must be shut down completely, then restarted, to effect a change in this parameter."<br /><br />I guess they stay at default for us. :)digitalpointhttps://www.blogger.com/profile/13595987087265745150noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-8475356635768295752015-03-06T23:16:08.255+01:002015-03-06T23:16:08.255+01:00Checkpoint parameters are important to set properl...Checkpoint parameters are important to set properly.<br />There are now 4 ones to consider:<br />MinDiskWriteSpeed<br />MaxDiskWriteSpeed<br />MaxDiskWriteSpeedOtherNodeRestart<br />MaxDiskWriteSpeedOwnRestart<br /><br />The old parameter<br />DiskCheckpointSpeed is deprecated in 7.4.<br /><br />The speed in normal operation goes between MinDiskWriteSpeed and MaxDiskWriteSpeed and<br />in a node restart all nodes speed up to at most<br />MaxDiskWriteSpeedOtherNodeRestart whereas the starting node can go all the way up to MaxDiskWriteSpeedOwnRestart.<br /><br />Most of the speedups that makes one jump a few wait states requires that the master node (the oldest node) is on 7.4. So this won't be the case in a rolling upgrade from 7.3 to 7.4. So hopefully the next one should be faster, but still important to set the config parameter above to appropriate numbers.Mikael Ronstromhttps://www.blogger.com/profile/07134215866292829917noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-10620105382389559992015-03-06T20:20:02.853+01:002015-03-06T20:20:02.853+01:00Looking forward to the blog entry about checkpoint...Looking forward to the blog entry about checkpoint speed config parameters.<br /><br />Side note - for me, 60GB data nodes were taking about 50 minutes to restart under 7.3. Under 7.4, the first one is up to 70 minutes. Maybe a rolling restart from 7.3 to 7.4 is slower (hopefully that's what it is)? {shrug}digitalpointhttps://www.blogger.com/profile/13595987087265745150noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-48247788672412605232015-03-04T23:26:19.193+01:002015-03-04T23:26:19.193+01:00No information is available during a cluster resta...No information is available during a cluster restart for the reason you mentioned. However during a node restart information about ongoing node restarts is available.Mikael Ronstromhttps://www.blogger.com/profile/07134215866292829917noreply@blogger.comtag:blogger.com,1999:blog-14455177.post-75824917116992338012015-03-04T23:14:14.999+01:002015-03-04T23:14:14.999+01:00Is any of the info available when starting up an e...Is any of the info available when starting up an entire cluster through the ndbapi? The mysql api nodes won't be able to connect until the cluster is started.Kadaanhttps://www.blogger.com/profile/11826029508824702882noreply@blogger.com