Almost every production deployment depends on some form of secrets, for database access, initial admin accounts or access to external APIs. There are multiple ways to hand secrets to applications, but each makes a unique tradeoff between maintainability, security and runtime complexity. Understanding that choice is essential for developing robust software.
Scope
Although most of the details explained in this article apply to configuration as a whole, the specific focus is on secrets, aka configuration values that nobody else should be aware of. These include access tokens, API keys, authentication credentials for database and the likes. Secrets are a subset of configuration parameters that live in your configuration, but have a much higher value to attackers, as they offer more potential for abuse or privilege escalation.
Command-line arguments and flags
Practically any programming language will have builtin standard library support for parsing and processing command-line arguments and flags. Using these for configuration is common for terminal-based applications, and many administrators expect tools to provide some standard flags like -h/--help for help output, -v/--verbose for increased debug information or --version.
This implicit expectation allows tight coupling of configuration options with documentation in help output, so no external links or side channel documentation (websites, man pages) have to be shipped with the program.
That said, they are a read-only part of the program and cannot be changed at runtime; changing command-line parameters requires stopping the current process and restarting it with new arguments.
Expressing anything beyond simple key-value pairs through runtime arguments adds verbosity or parsing overhead.
Lists of dynamic-length items require repetition or non-standard value formatting:
./myprogram --mount /data/drive1 --mount /dev/drive2 --mount /dev/drive3 ...
./myprogram --mount /drive1,/drive2,/drive3,...Hierarchies or nested value grouping rely on extremely long flag names, argument order or custom traversal expressions:
./myprogram --security--backup--reset-key "mykey"
./myprogram --security --backup --reset-key "mykey"
./myprogram --security "backup.reset-key=mykey"It is also a bad fit for long text values, like base64 encoded cryptographic keys.
From a security perspective, runtime arguments are ill-suited for anything requiring protection, as they are visible for anyone on the machine in process lists. If a user "sam" starts a program as ./myprogram --key 123, then user "bob" can see it including the --key flag and value when looking at the output of ps aux. Even special users like root are not protected from this unprivileged process exposure.
Environment variables
Environment variables are extremely easy to parse in any programming language, but are stored in process memory instead of readable metadata.
Just as cli arguments, they cannot be changed externally at runtime (although a process can modify its own environment variables), but have even more difficulty expressing complex configuration hierarchies. Environment variables cannot be specified multiple times (same name quietly overrides previous values) and do not maintain any reliable sorting order.
Coupling documentation with this type of configuration is impossible, so it requires external documentation through websites, man pages or bundling with text/markdown files.
Environment variables are not readable outside the owning process for users other than the user owning the process and root, but are implicitly inherited in child processes by default. They are a safe choice for simple configurations and secrets that do not require dynamic mapping or hierarchical structures.
Configuration files
As one of the oldest forms of configurations, config files have remained an excellent tradeoff between usability and security.
Being an external file, they may need mounting into isolated namespaces or containers, but can change at runtime without restarting the process. Many daemons and background services rely on this feature, allowing them to "reload" only the configuration by re-reading the file without restarting the entire process, allowing open file or network operations to remain uninterrupted.
Config files can be generated when starting without one, including proper documentation for every possible value, tightly coupling documentation with the program. File contents are not standardized, leading to a wide range from simple key=value syntax like dotenv to more advanced formats like yaml or toml.
Depending on format, extremely dynamic or deeply nested configurations can be expressed with ease, but parsing is non-trivial and requires third-party software or development overhead for the parsing logic.
The security of configuration files depends solely on the file's read permissions. Relying on a file adds complexity to deployments: Operators must now run the application as a user with read access to the config file while others do not, and unexpected errors (mounting issues, drive failures, permission changes) can leak into the application at runtime.
Security-hardened alternatives
More advanced injection methods rely on a sidecar process that manages secrets for the host and injects them into select processes on-demand.
They look and feel different, but rely on one of these two basic approaches:
They pass a file descriptor to a unix socket, in-memory file or unix pipe to the application, which it may use to read config and secrets once, then closes the file handle, leaving no trace of the data outside the two programs.
They pass a reference to shared memory that was preloaded with the secrets, often encrypted, to the application on start, which it may then read and overwrite itself.
Most sidecar processes use abstractions and more complex inter-process communication (IPC) protocols, but rely on one of these sharing mechanisms internally.
All of these mechanisms are non-trivial to implement and require a secondary secret manager process, typically bound to a specific secret manager product. The resulting vendor lock-in means a change in security profile or secret management will require critical code adjustments for all applications.
Documentation can not be directly coupled into these mechanisms, and support for dynamic runtime changes or nested configuration parameters varies per implementation.
They provide the best security for the highest friction and failure potential, relying on features that may not be available in containers/namespaces (shared memory) and requiring host dependencies (sidecar process) that can't easily be ported between machines.
For this reason, this solution is mostly reserved for highly sensitive environments that can afford extra cost in development and maintenance complexity for maximum security.
What about secret managers?
Secret managers increase the protection of secrets during delivery, but not at the host level. They have to use one of the approaches outlined in this article for the "last mile" of delivery, specifically the process of injecting it into an application. Using a secret manager adds secret protection at rest and during network transfer, centralized storage, and enables audit logging, credential rotation and ephemeral access tokens - none of which affect the final injection process per host.
When to pick what
Choosing the best fitting approach for your use case can be broken down into a short checklist:
Processes that do not need runtime reloading support or protection from other system users can rely on command-line arguments. These are mostly terminal tools and applications for desktops or single-user workstations.
Programs with only simple configuration that can be expressed as key=value pairs, especially when running inside containers, will find most value by using environment variables
When needing more configuration structure/nesting, persistent configuration between restarts or the ability to reload without restarting the process, good old config files remain a solid choice
If security at all costs is paramount and you are willing to stick with external dependencies like secret managers and agent sidecar processes per host, there is no way around shared memory or
fdpassing.