Fork me on Github Edit me on Github
Blended Zio Core

blended-zio-core #

Functionality that is required by all blended containers.

Configuring Blended Containers #

As outlined here all modules that require configuration should be able to use external configuration files containing place holders to specify lookups from environment variables or resolve encrypted values.

For example, the configuration for an LDAP service might be:

{
  url             : "ldaps://ldap.$[[env]].$[[country]]:4712"
  systemUser"     : "admin"
  systemPassword" : "$[(encrypted)[5c4e48e1920836f68f1abbaf60e9b026]]"
  userBase"       : "o=employee"
  userAttribute"  : "uid"
  groupBase"      : "ou=sib,ou=apps,o=global"
  groupAttribute" : "cn"
  groupSearch"    : "(member={0})"
}

The ZIO ecosystem has a library called zio-config which supports different sources such as property files, HOCON, YAML or even the command line. At the core of the library are ConfigDescriptors which can be used to read the config information into config case classes. The descriptors are also used to generate documentation for the available config options or reports over the configuration values used within the application.

Following the advice from the zio-config library author on discord, we introduce a LazyConfigString as follows:

 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
sealed abstract case class LazyConfigString(value: String)

object LazyConfigString {

  import ConfigDescriptorAdt._

  final case class Raw(raw: String) {
    def evaluate(
      ctxt: Map[String, String]
    ): ZIO[StringEvaluator.StringEvaluator, EvaluatorException, LazyConfigString] = for {
      se  <- ZIO.service[StringEvaluator.Service]
      res <- se.resolveString(raw, ctxt)
    } yield (new LazyConfigString(res) {})
  }

  val configString: ConfigDescriptor[Raw]               =
    Source(ConfigSource.empty, LazyConfigStringType) ?? "lazyly evaluated config string"
  def configString(path: String): ConfigDescriptor[Raw] = nested(path)(configString)

  private case object LazyConfigStringType extends PropertyType[String, LazyConfigString.Raw] {
    def read(v: String): Either[PropertyType.PropertyReadError[String], LazyConfigString.Raw] = Right(Raw(v))
    def write(a: LazyConfigString.Raw): String                                                = a.raw
  }
}

Essentially we define a class LazyConfigString, which instances will eventually hold the resolved config value. Making the class sealed and abstract ensures that new instances can only bo created from within the companion object.

Within the companion object the case class Raw can be instantiated with Strings read from the config sources. Also, within this class the evaluate method holds the effect describing the resolution of the raw config string to a real value. Essentially we are deferring the resolution to a StringEvaluator service.

At last we need to provide a config descriptor for LazyConfigStrings, so that the generated documentation will reflect that the config values are subject to lazy evaluation.

Using the LazyConfigString, we can define the LDAPConfig as follows:

74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
object LDAPConfig {

  import LazyConfigString._

  def desc: ConfigDescriptor[LDAPConfig] = (
    configString("url") ?? "The url to connect to the LDAP server" |@|
      configString("systemUser") |@|
      configString("systemPassword") |@|
      configString("userBase") |@|
      configString("userAttribute") |@|
      configString("groupBase") |@|
      configString("groupAttribute") |@|
      configString("groupSearch")
  )(LDAPConfig.apply, LDAPConfig.unapply)
}

case class LDAPConfig(
  url: LazyConfigString.Raw,
  systemUser: LazyConfigString.Raw,
  systemPassword: LazyConfigString.Raw,
  userBase: LazyConfigString.Raw,
  userAttribute: LazyConfigString.Raw,
  groupBase: LazyConfigString.Raw,
  groupAttribute: LazyConfigString.Raw,
  groupSearch: LazyConfigString.Raw
)

To access a config value, a layer with a StringEvaluator must be referenced:

51
52
53
54
55
56
57
58
59
  private val desc: ConfigDescriptor[LDAPConfig] = LDAPConfig.desc
  private val ctxt: Map[String, String]          = Map("user" -> "ADMIN", "env" -> "dev", "country" -> "es")

  private def simpleEval(src: ConfigSource) = testM("Evaluate a simple config map")(
    (for {
      cfg <- ZIO.fromEither(read(desc.from(src)))
      pwd <- cfg.systemPassword.evaluate(ctxt)
    } yield assert(pwd.value)(equalTo("blended"))).provideLayer(evalLayer)
  )

With the code above, zio-config will generate the following report in markdown format:

Configuration Details

FieldName Format Description Sources
all-of

Field Descriptions

FieldName Format Description Sources
url primitive lazyly evaluated config string, The url to connect to the LDAP server
systemUser primitive lazyly evaluated config string
systemPassword primitive lazyly evaluated config string
userBase primitive lazyly evaluated config string
userAttribute primitive lazyly evaluated config string
groupBase primitive lazyly evaluated config string
groupAttribute primitive lazyly evaluated config string
groupSearch primitive lazyly evaluated config string

Evaluate simple string expressions #

Lazyly evaluated string expressions are simple expressions as defined here:

 8
 9
10
11
sealed trait StringExpression
case class SimpleExpression(value: String)                                      extends StringExpression
case class SequencedExpression(parts: Seq[StringExpression])                    extends StringExpression
case class ModifierExpression(modStrings: Seq[String], inner: StringExpression) extends StringExpression

The notable piece here is the ModifierExpression, which has the form

$[modifier*[StringExpression]]

A modifier expression contains an inner expression and evaluation will be from the innermost expression outwards. After resolving the inner expression, zero or more modifiers will be applied to the resolved value for a given context. The context is a simple Map[String, String] and the normal resolution simply maps the resolved expression to the corresponding value in the map.

For example, the expression $[[foo]] with the context map Map("foo" -> "bar") will yield "bar".

Modifiers will be applied to the value resolved from the context map, for example with the context map from above

$[(upper)[foo]] => "BAR"
$[(left:2)[foo]] => "ba"

Modifiers are specified as:

 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
trait Modifier {
  def name: String
  def op(s: String, p: String): ZIO[Any, Throwable, String]

  def lookup: Boolean = true

  final def modifier(
    s: String,
    p: String
  ): ZIO[Any, ModifierException, String] = (for {
    input <- ZIO.fromOption(Option(s))
    param  = Option(p).getOrElse("")
    mod   <- op(input, param)
  } yield mod).mapError {
    case me: ModifierException => me
    case t: Throwable          => new ModifierException(this, s, p, t.getMessage)
    case _                     => new ModifierException(this, "", p, "Segment can't be null")
  }
}

A modifier implementation can override lookup to avoid that the value resolved from the inner expression will be used to look up the final value from the context map.

The EncryptModifier does that, so that the decryption will be applied to the string resolved from the inner expression.

Simple crypto service #

The EncryptModifier is defined as

85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
object EncryptedModifier {

  def create: ZIO[CryptoSupport.CryptoSupport, Nothing, Modifier] = for {
    cs <- ZIO.service[CryptoSupport.Service]
    mod = new Modifier {
            override def name: String = "encrypted"

            override def lookup = false

            override def op(s: String, p: String): ZIO[Any, Throwable, String] = (for {
              res <- cs.decrypt(s)
            } yield (res))
          }
  } yield mod
}

It relies on a crypto service available within the ZIO environment and simply delegates the resolution to the decrypt method of that service.

The crypto service is defined as

35
36
37
38
  trait Service {
    def encrypt(plain: String): ZIO[Any, CryptoException, String]
    def decrypt(encrypted: String): ZIO[Any, CryptoException, String]
  }

The default implementation can be instantiated with a password, for convenience the code also contains a default password. The password can also be provided via a file. Essentially, the provided password is used to generate a key that is then used to create an instance of a CryptoService which simply wraps some Crypto methods from Java:

 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
final private class DefaultCryptoSupport(key: Key, alg: String) {

  def decrypt(encrypted: String): ZIO[Any, CryptoException, String] = for {
    bytes     <- string2Byte(encrypted)
    ciph      <- cipher(Cipher.DECRYPT_MODE)
    decrypted <- ZIO.effect(ciph.doFinal(bytes.toArray)).refineOrDie { case t => new CryptoFrameworkException(t) }
  } yield (new String(decrypted))

  def encrypt(plain: String): ZIO[Any, CryptoException, String] = for {
    ciph  <- cipher(Cipher.ENCRYPT_MODE)
    bytes <- ZIO.effect(ciph.doFinal(plain.getBytes())).refineOrDie { case t => new CryptoFrameworkException(t) }
    res   <- byte2String(bytes.toSeq)
  } yield res

  private def cipher(mode: Int): ZIO[Any, CryptoException, Cipher] =
    ZIO.effect { val res = Cipher.getInstance(alg); res.init(mode, key); res }.refineOrDie { case t =>
      new CryptoFrameworkException(t)
    }

  private def byte2String(a: Seq[Byte]): ZIO[Any, Nothing, String] =
    ZIO.collectPar(a)(b => ZIO.succeed(Integer.toHexString(b & 0xff | 0x100).substring(1))).map(_.mkString)

  private def string2Byte(s: String, orig: Option[String] = None): ZIO[Any, CryptoException, Seq[Byte]] = {
    val radix: Int = 16

    (s match {
      case ""                               => ZIO.succeed(Seq.empty)
      case single if (single.length() == 1) =>
        ZIO.effect(Seq(Integer.parseInt(single, radix).toByte)).refineOrDie { case _: NumberFormatException =>
          new InvalidInputException(orig.getOrElse(s))
        }
      case s                                =>
        string2Byte(s.substring(2), Some(orig.getOrElse(s)))
          .map(rest => Seq(Integer.parseInt(s.substring(0, 2), radix).toByte) ++ rest)
    })
  }
}

Using the services #

To use the services resolving config string, a layer with all required services must be provided:

32
33
34
35
36
37
38
39
40
41
42
  private val logSlf4j = Slf4jLogger.make((_, message) => message)

  private val cryptoDefault: ZLayer[Any, Nothing, CryptoSupport.CryptoSupport] =
    CryptoSupport.default.orDie

  private val mods: ZIO[Any, Nothing, Seq[Modifier]] = EncryptedModifier.create.provideLayer(cryptoDefault).map { em =>
    Seq(UpperModifier, LowerModifier, em)
  }

  private val evalLayer: ZLayer[Any, Nothing, StringEvaluator.StringEvaluator] =
    logSlf4j >>> StringEvaluator.fromMods(mods)

This layer can be provided to an effect by the means of provideLayer

51
52
53
54
55
56
57
58
59
  private val desc: ConfigDescriptor[LDAPConfig] = LDAPConfig.desc
  private val ctxt: Map[String, String]          = Map("user" -> "ADMIN", "env" -> "dev", "country" -> "es")

  private def simpleEval(src: ConfigSource) = testM("Evaluate a simple config map")(
    (for {
      cfg <- ZIO.fromEither(read(desc.from(src)))
      pwd <- cfg.systemPassword.evaluate(ctxt)
    } yield assert(pwd.value)(equalTo("blended"))).provideLayer(evalLayer)
  )

Finally, the config can be resolved from a config source created from a Map with ConfigSource.fromMap:

74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
object LDAPConfig {

  import LazyConfigString._

  def desc: ConfigDescriptor[LDAPConfig] = (
    configString("url") ?? "The url to connect to the LDAP server" |@|
      configString("systemUser") |@|
      configString("systemPassword") |@|
      configString("userBase") |@|
      configString("userAttribute") |@|
      configString("groupBase") |@|
      configString("groupAttribute") |@|
      configString("groupSearch")
  )(LDAPConfig.apply, LDAPConfig.unapply)
}

case class LDAPConfig(
  url: LazyConfigString.Raw,
  systemUser: LazyConfigString.Raw,
  systemPassword: LazyConfigString.Raw,
  userBase: LazyConfigString.Raw,
  userAttribute: LazyConfigString.Raw,
  groupBase: LazyConfigString.Raw,
  groupAttribute: LazyConfigString.Raw,
  groupSearch: LazyConfigString.Raw
)