I deleted my persistent settings stream that contained my domains!

Scenario:  You accidentally deleted your original persistent settings stream, which is why the Swarm nodes are reporting "Failed to read cluster persistent settings".

This stream is merely a text file with settings information and has nothing to do with the _administrators stream. Within the persistent settings stream is a setting called "domains", which contains a list of domains and the UUID of the domain stream. This is the list that is presented on the admin console and gets updated when a new domain is created.

If a new persistent settings stream is created after a domain already exists, the persistent settings stream does not know about that domain.

What we can do is add the domain into the persistent settings stream manually with an APPEND.

If you info the settings stream, you should see the line: domains=[]

$ curl -L --anyauth -u "admin:ourpwdofchoicehere" --location-trusted http://192.168.99.110/f77cbc84973733b0910925e0c8a7cb3d?alias

loghost=192.168.99.99
powerCap=100 
powerSavingMode=1 
logport=514 
repMulticastFrequency=1 
decoderSettings={} 
loglevel=20 
sleepAfter=7200 
wakeAfter=28800 
domains=[] 
volumeRecoverySuspend=0

This is assuming there have not been any new domains created since the creation of the new settings stream.

What we want to do is APPEND the information for the domain that you want displayed in the console.

The format is:
domains=[('domain name', 'domain alias UUID')]

So first, info the domain stream itself to get the UUID. 

On my test cluster, the domain is called my.test.domain. 

$ curl -IL --anyauth -u "admin:password" --location-trusted "http://192.168.99.110?domain=my.test.domain"

Among the response headers is the Castor-System-Alias: 
Castor-System-Alias: ac088276264abc849d84e9ccd1ddcbae

So now I need to append the domain name and UUID to my persistent settings stream: 

$ curl -L -X APPEND --data-binary "domains=[('my.test.domain', 'ac088276264abc849d84e9ccd1ddcbae')]" --anyauth -u "admin:ourpwdofchoicehere" --location-trusted http://192.168.99.110/f77cbc84973733b0910925e0c8a7cb3d?alias --post301 -v

Now when you read the persistent settings stream back, you should see the new "domains" line at the end of the file, but it may not have been picked up by the nodes yet. Reboot the nodes and the new domain setting will be picked up if this is the case.