Collection Overview<\/h2>\n
This text is Half 2.3 of a multi-part sequence: “Improvement of system configuration administration.”<\/p>\n
The whole sequence:<\/p>\n
- \n
- Introduction<\/li>\n
- Migration finish evolution\n
- \n
- Working with secrets and techniques, IaC, and deserializing knowledge in Go<\/li>\n
- Constructing the CLI and API<\/li>\n
- Dealing with unique configurations and related templates<\/li>\n<\/ol>\n<\/li>\n
- Efficiency consideration<\/li>\n
- Abstract and reflections<\/li>\n<\/ol>\n
Unique Hosts Configuration<\/h2>\n
Goal and Necessities<\/h3>\n
The minimal configuration unit for the brand new SCM is a bunch group that performs the same operate within the infrastructure. In different phrases, we are able to outline this as a task for the hosts. Every function has an setting that describes the host’s belonging to a devoted set for various functions. For instance, a task known as backend<\/strong> can have three environments: prod, stage,<\/strong> and dev<\/strong> within the venture myproj<\/strong>. Consequently, these names outline the hostgroup<\/a>:<\/p>\n
- \n
- myproj-dev-backend<\/li>\n
- myproj-stage-backend<\/li>\n
- myproj-prod-backend<\/li>\n<\/ul>\n
Every host group consists of a sure variety of hosts, which may generally embrace only one host. Nonetheless, an essential precept that has lengthy been established is that we can not configure objects smaller than a hostgroup utilizing the brand new SCM. This strategy is logical, as host teams are used to allow a number of hosts for horizontal scaling and guarantee excessive availability for every service. Generally, such hosts share the same configuration, and we operated beneath the paradigm that if we would have liked to configure just one host with no hostgroup, we have been doubtless doing one thing fallacious.<\/p>\n
Nonetheless, we ultimately concluded that there are situations the place extra detailed configurations are mandatory than these relevant to a bunch group. There are additionally many instances the place we have to apply configurations to a number of hosts inside a specific hostgroup. A particular case of this would possibly contain making use of configurations in a canary method to at least one or a couple of hosts to reduce the affect on the remainder of the hostgroup.<\/p>\n
Within the early phases of growing the brand new SCM, we used a workaround involving stopping the SCM agent or locking updates on some hosts. This allowed us to check deployments on smaller subsets of the host group earlier than making use of the configuration to the complete group. Nonetheless, this strategy has a number of apparent points:<\/p>\n
- \n
- Testing subsets of servers prevents them from receiving new configuration updates, affecting not simply the half that’s at the moment being examined.<\/li>\n
- Testing subsets of servers stops checking the verification of the state of the configuration. For example, the SCM agent verifies that the mandatory companies are operating.<\/li>\n
- These handbook operations don’t enhance general automation.<\/li>\n<\/ul>\n
In accordance with the above, the next necessities for unique configuration have been established:<\/p>\n
- \n
- The flexibility to create configurations per host or for a couple of hosts<\/li>\n
- The flexibility to create particular configuration blocks that don’t intervene with different modules<\/li>\n
- The flexibility to regulate unique configurations remotely to satisfy CI\/CD wants<\/a><\/li>\n<\/ol>\n
Doable Methods to Implement This<\/h3>\n
After we began analyzing potential methods to implement unique configurations, we confronted a key query: the place ought to we retailer the configurations? Primarily, we had two choices.<\/p>\n
Firstly, it may be saved as a easy file, much like a hostgroup YAML file. This file ought to have a better precedence in comparison with the group configuration. Nonetheless, this strategy shouldn’t be very helpful as a result of it doesn’t deal with the third requirement. It can’t be used from CI with out workarounds. Though we in the end didn’t select this feature, it grew to become possible to make use of it not directly later, after enabling templating for YAML recordsdata:<\/p>\n
\n\n\n{{- if eq .IP \"10.10.10.10\" }}\nconfiguration for this host solely\n{{- else }}\nconfiguration for different hosts in hostgroup\n{{- finish }}<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nSecondly, we thought of utilizing an API router for unique configuration management. On this case, we are able to management configurations remotely with correct authorization. For the reason that API makes use of Consul to retailer knowledge, the configuration acquired by the API may be saved in Consul by the SCM, which may then make the most of these keys within the scheduler to construct the entire host configuration. This answer glad all necessities.<\/p>\n
In follow, implementing this was comparatively simple \u2014 we added this logic on the CLI stage of the deployment device, which processes requests for particular person hosts. The API saves this knowledge in a devoted path: unique\/host\/{ip}<\/strong>. The scheduler then connects to this path when merging configurations, permitting for various configurations for every host. The merged configuration of hosts takes priority, and any beforehand declared configurations may be overwritten by these particular settings.<\/p>\n
We started utilizing this configuration to implement rollout deployment logic much like that of Kubernetes for non-Docker tasks. The entire logic was built-in on the CLI stage, the place the device acknowledges the variety of hosts in hostgroups, performs deployments throughout the desired variety of hosts, and waits for them to run.<\/p>\n
Implementation within the Code<\/h3>\n
A route
api\/v3\/unique<\/code> has been added to the API that gives a CRUD interface. So as to add an unique configuration, ship an HTTP POST request with the next construction:<\/p>\n\n\n\nkind ExclusivePostData struct {\n Part string `json:\"part\"`\n Hosts []string `json:\"hosts\"`\n Ttl int `json:\"ttl\"`\n Knowledge string `json:\"knowledge\"`\n Instantly bool `json:\"instantly\"`\n}<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nThe place:
part<\/code> is the context identify for the ensuing JSON despatched to the chosen hosts.knowledge<\/code> is the JSON containing the configuration knowledge.hosts<\/code> are the chosen hosts.ttl<\/code> is the overall time to retailer this key in Consul.instantly<\/code> specifies whether or not to deploy instantly utilizing the push strategy.<\/p>\nAs soon as the API router has acquired all the mandatory knowledge, it saves the important thing in Consul:<\/p>\n
\n\n\nconsul:~$ consul kv get -recurse unique\/10.9.2.116\/\nunique\/10.9.2.116\/packages:{\"iftop\":{}}<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nAs talked about earlier, we would have liked to replace the configuration generator. We added one other merge to counterpoint this per-host configuration:<\/p>\n
\n\n\n\/\/ merge configuration tree with consul unique knowledge\nerr = mergo.Map(&DefaultConf, ConsulExclusive, mergo.WithOverride, mergo.WithAppendSlice)\nif err != nil {\n logger.GenConfLog.Println(\"mergo Map failed with DefaultConf and ConsulExclusive\")\n return map[string]interface{}{}, err\n}<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nThe
mergo.WithAppendSlice<\/code> flag is especially helpful on this case, because it permits us so as to add info to a slice as a substitute of changing it.<\/p>\nTwo strategies, GET and DELETE, have been applied for creating and deleting unique configurations.<\/p>\n
Easy methods to Use This in Follow<\/h3>\n
The
unique<\/code> command has been added to the CLI device, enabling it to function with the unique API route on the SCM API.<\/p>\n\n\n\n~$ cli unique --help\nNAME:\n cli unique - Get\/push\/delete unique configuration for host(s)\n\nUSAGE:\n cli unique command [command options] [arguments...]\n\nCOMMANDS:\n push push unique configuration for host(s)\n get get present unique configuration for host(s)\n delete delete unique configuration for host(s)\n\u00a0 \u00a0assist, h \u00a0Reveals a listing of instructions or assist for one command<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nOne instance of how unique configurations may be useful is in canary deployments. The next script was applied to be used in CI:<\/p>\n
- \n
- Sending an unique configuration to the SCM Nginx module to take away one upstream host from the load balancing pool<\/li>\n
- Sending an unique configuration to the SCM Docker module to replace the Docker picture on the host<\/li>\n
- Utilizing numerous checkers to check the appliance for updates and well being<\/li>\n
- Eradicating the unique configuration from the SCM after which enabling site visitors for the up to date backend<\/li>\n
- Repeating this for every host within the subset<\/li>\n<\/ol>\n
Disadvantages of Utilizing Unique Configurations<\/h3>\n
The primary drawback of unique configurations is their opacity; customers could overlook that they’ve been created. Whereas there’s a TTL performance that permits configurations to revert after a specified time, the use case for TTL is primarily appropriate for testing.<\/p>\n
Metrics and Logs<\/h2>\n
Pull SCM brokers can\u2019t present detailed info when functioning in methods aside from by logs and metrics. We applied completely different loggers at each the agent and API ranges. This enchancment enhanced observability and debugging through the growth of our SCM and stays helpful now that energetic growth is completed.<\/p>\n
Many loggers write to a number of recordsdata primarily based on their space. Any references to the file module are logged in ‘file.log’, service-related logs go to ‘service.log’, and package-related logs are written to ‘package deal.log’. What in regards to the different modules? Sure, every module has its personal designated log file, similar to:<\/p>\n
- \n
- aerospike.log<\/li>\n
- nginx.log<\/li>\n
- clickhouse.log<\/li>\n
- and many others.<\/li>\n<\/ul>\n
On the API stage, configuration mergers write logs to their respective recordsdata. Equally, on the SCM agent stage, parsers additionally log to the related recordsdata. The implementation for that is fairly easy:<\/p>\n
\n\n\nfunc LogInit() {\n PackagesLog = CreateLog(conf.LogPackagesPath, \"packages\")\n ServicesLog = CreateLog(conf.LogServicesPath, \"companies\")\n VaultLog = CreateLog(conf.LogVaultPath, \"vault\")\n AerospikeLog = CreateLog(conf.LogAerospiketPath, \"aerospike\")\n PKILog = CreateLog(conf.LogPKIPath, \"pki\")\n DiskLog = CreateLog(conf.LogDiskPath, \"disk\")\n UserLog = CreateLog(conf.LogUserPath, \"consumer\")\n}\n\nfunc CreateLog(LogPath string, Part string) (*log.Logger){\n file, err := os.OpenFile(Path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0660)\n if err != nil {\n log.Fatalln(\"Didn't open log file\", Path, \":\", err)\n }\n retLogger := log.New(file, \": \", log.Ldate|log.Ltime|log.Lshortfile)\n\n SyslogPrefix := os.Getenv(\"SYSLOG_PREFIX\")\n SyslogAddr := os.Getenv(\"SYSLOG_ENDPOINT\")\n SyslogProto := os.Getenv(\"SYSLOG_PROTO\")\n if SyslogPrefix != \"\" && Part != \"\" {\n TagName := SyslogPrefix + \"-\" + Part\n\n if SyslogAddr != \"\" {\n if SyslogProto == \"\" {\n SyslogProto = \"udp\"\n }\n\n syslogger, err := syslog.Dial(SyslogProto, SyslogAddr, syslog.LOG_INFO, TagName)\n if err != nil {\n log.Fatalln(err)\n }\n\n retLogger.SetOutput(syslogger)\n } else {\n syslogger, err := syslog.New(syslog.LOG_INFO, TagName)\n if err != nil {\n log.Fatalln(err)\n }\n\n retLogger.SetOutput(syslogger)\n }\n }\n\n return retLogger\n}<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nSyslog serves in its place approach to monitor points throughout the infrastructure. Syslog messages are acquired by the native rsyslog and despatched to ElasticSearch.<\/p>\n
Essentially the most difficult side is logging the variations in recordsdata on account of probably delicate info. These logs are saved solely on the filesystem in ‘recordsdata.log’, with permissions set to 0600, accessible solely to the foundation consumer.<\/p>\n
With the push scheme, it’s potential to return HTTP responses indicating the variations to the consumer that initiated the deployment:<\/p>\n
\n\n\nfunc FilesParser(ApiResponse map[string]interface{}, ClientResponse map[string]interface{}) {\n FilesMap, Resp, ActiveParser := ParserGetData(ApiResponse, ClientResponse, \"recordsdata\")\n...\n for FileName, FileOptions := vary FilesMap {\n...\n GenFile, err := GenTmpFileName(FileName)\n if err != nil {\n logger.FilesLog.Println(\"Can't write non permanent file:\", GenFile, \"with error\", err)\n Resp[FileName] = map[string]interface{}{\n \"state\": \"error\",\n \"error\": err.Error(),\n }\n metrics.FileError.Inc()\n proceed\n }\n\n err = ioutil.WriteFile(GenFile, knowledge, filemode)\n if err != nil {\n Resp[FileName] = map[string]interface{}{\n \"state\": \"error\",\n \"error\": err.Error(),\n }\n logger.FilesLog.Println(\"Can't write to file:\", FileName, \":\", err)\n metrics.FileError.Inc()\n proceed\n }\n\n err, end result = CompareAndCopyFile(FileName, GenFile, filemode, FileUser, FileGroup) {\n if err == nil {\n if end result {\n Resp[FileName] = map[string]interface{}{\n \"state\": \"modified\",\n \"diff\": CalculatedDiff,\n }\n metrics.FileDeployed.Inc()\n }\n } else {\n Resp[FileName] = map[string]interface{}{\n \"state\": \"error\",\n \"error\": err.Error(),\n }\n metrics.FileError.Inc()\n proceed\n }\n }\n...<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nThis demonstrates a functioning suggestions system that gives responses to the push brokers. Just like Ansible<\/a> or SaltStack, this method allows the power to watch the outcomes of deployments on hosts and spotlight errors. This instance additionally reveals how the metrics work. We’ve got quite a lot of metrics; some operate solely on the agent stage, whereas others function on the API\/scheduler stage.<\/p>\n
Relating to the metrics, we measure the variety of service restarts and reloads, file modifications, tried package deal installations, and extra. Consequently, we are able to monitor infrastructure-level points associated to package deal managers and deployment failures. For instance, the next graph illustrates issues with package deal installations throughout the infrastructure:<\/p>\n
![\"Problems](\"https:\/\/dz2cdn1.dzone.com\/storage\/temp\/18457780-1749295721561.jpeg\")
<\/p>\nOne other instance measures the final restarts of companies over the month attributable to our SCM agent:<\/p>\n
![\"Last](\"https:\/\/dz2cdn1.dzone.com\/storage\/temp\/18455627-1749273156739.jpeg\")