{"componentChunkName":"component---src-templates-default-template-tsx","path":"/guides/rl-user-guide/","result":{"data":{"asciidoc":{"id":"f552a231-9b35-5f12-b14f-0863d3bb6782","html":"
\n
Table of Contents
\n\n
\n
\n

Introduction

\n
\n
\n

The AI part of HIRO is based on Reinforcement Learning algorithms.

\n
\n
\n

Reinforcement Learning is an aspect of Machine learning where an agent learns to behave in an environment, by performing certain actions and observing the rewards/results which it get from those actions.

\n
\n
\n

The algorithm (agent) evaluates a current situation (state), takes an action, and receives feedback (reward) from the environment after each act. Positive feedback is a reward (in its colloquial meaning), and negative feedback is punishment for making a mistake. Reinforcement Learning algorithms learn how to act best through many attempts and failures.

\n
\n
\n

The current implementation of Reinforcement Learning in HIRO uses so-called Monte Carlo methods.

\n
\n
\n

To make Reinforcement Learning working successfully, we need to follow some basic rules when creating issues and writing KIs.

\n
\n
\n
\n
\n

Concepts and Notations

\n
\n
\n

First, lets introduce some RL-specific concepts and notations:

\n
\n
\n
\n\"Reinforcement\n
\n
\n
\n
    \n
  1. \n

    An Episode in HIRO is called an Issue/Task.

    \n
    2. \n
  2. \n

    HIRO is an Agent.

    \n
    3. \n
  3. \n

    To resolve an issue, HIRO is interacting with an Environment.

    \n
    4. \n
  4. \n

    A Time step consists of choosing and executing a single KI.

    \n
    5. \n
  5. \n

    Therefore at each time step HIRO is taking some Action (executing some KI).

    \n
    6. \n
  6. \n

    At each time step HIRO is observing an environment. This particular observation at a particular time step is called a State.

    \n
    7. \n
  7. \n

    Once a KI is executed, HIRO can obtain some Reward (numerical value). It is not necessary that a reward is received at every time step.

    \n
    8. \n
  8. \n

    Once the reward is obtained, HIRO can (but not necessarily) observe a New State. When the issue is resolved in this step, this will be the Final state.

    \n
    9. \n
  9. \n

    HIRO takes steps until the end of the episode/issue is reached.

    \n
    10. \n
  10. \n

    The main goal is to maximize the total reward and to make sure we take the most promising steps (execute the best possible KIs).

    \n
    11. \n
\n
\n
\n

A short description of this process could look like this:

\n
\n
\n
\n
`state => action => reward => nextstate => action => reward => nextstate => … => action => reward => finalstate`
\n
\n
\n
\n
\n
\n

Examples

\n
\n
\n

An example to illustrate this is the famous Tower of Hanoi.

\n
\n
\n

Each disk can use one particular rod. That will be the State.\nWe can represent the state as a string:

\n
\n
\n
\n\"Hanoi\n
\n
\n
\n

All three disks are currently using the first rod. This state can be represented as HanoiPos_1_1_1

\n
\n
\n
\n\"Hanoi\n
\n
\n
\n

Two disks are using the first rod and the third disk is using the second rod. This state can be represented as HanoiPos_1_1_2

\n
\n
\n

To train the system to finish the task as fast as possible, we assign a reward of \"-1\" for each time step.

\n
\n
\n

The Reward is crucial for our system to work correctly. We can obtain the reward directly from the environment or from the KI. In the last case we should increment the ogit/RL/totalReward variable by this local reward.

\n
\n
\n

ogit/RL/totalReward = ogit/RL/totalReward + local_reward

\n
\n
\n

To penalize some action we can use a negative reward:

\n
\n
\n

ogit/RL/totalReward = ogit/RL/totalReward + (-local_reward)

\n
\n
\n

The main idea is to encourage good actions/KIs and penalize bad actions/KIs, when it is possible. This approach is critical for HIRO and its Learning Service to learn optimal behavior. If you don_t provide any reward during the whole Issue/Episode, the system will not be able to learn anything useful from this episode.

\n
\n
\n

Another example is the MAZE environment:

\n
\n
\n
    \n
  • \n

    The Episode starts at some initial point and ends when we move to the exit.

    \n
    • \n
  • \n

    The Environment is the whole gridworld/maze, where we can navigate our agent from one state/position to another.

    \n
    • \n
  • \n

    The State can be represented as the current coordinates (X, Y) or just as a number 1,2,3..,N. We always store the current state internally.

    \n
    • \n
  • \n

    The Reward is a numerical value, which is given to our agent via KIs. In case of maze, after each time step HIRO gives the reward \"-1\" to our agent. By maximizing the total reward we are able to find the shortest path to the exit (e.g. our final reward is -100, which means it’s 100 steps, and that’s better than the reward -200, which is 200 steps). We store all the rewards obtained for each time step.

    \n
    • \n
\n
\n
\n
\n
\n

Workflow

\n
\n
\n

There are two main variables in the Reinforcement Learning workflow:

\n
\n
\n
    \n
  • \n

    ogit/RL/totalReward

    \n
    • \n
  • \n

    ogit/RL/state

    \n
    • \n
\n
\n
\n

Both are controlled via KIs.

\n
\n
\n

a) Use ogit/RL/totalReward when you need to encourage or penalize a particular KI.

\n
\n
\n

If you need to treat your KI as an agent_s step, you should explicitly add some reward inside of this KI.\ne.g. in the KI definition, inside of the DO section:

\n
\n
\n

ogit/RL/totalReward = ogit/RL/totalReward - 1

\n
\n
\n

This will add a reward of -1 to this action/KI, it therefore penalizes this KI.

\n
\n
\n

For example, in case of Maze or Tower of Hanoi, just use ogit/RL/totalReward = ogit/RL/totalReward - 1 for each KI so that you penalize extra steps taken and make HIRO to learn the shortest path.

\n
\n
\n

b) Use ogit/RL/state when you need to set or change your current state.

\n
\n
\n

Defining a meaningful state can be tricky. You need to think about what combination of environment parameters can be used as a state. For example, in a Maze environment a state can be just a X and Y coordinates (e.g. a string “22_15” which represents x=22 and y=15).\nIn Tower of Hanoi this will be HanoiPos_1_1_1, where each \"1\" represents a disk using the first rod.

\n
\n
\n

Example of a KI using the Reinforcement Learning system:

\n
\n
\n
\n
ki\n  id: \"HN3_R1\"\n  version: \"0@rs\"\n  description: \"Move disc 3 to position 1.\"\non\n  ogit/_id\nwhen\n  MakeMove == \"Hanoi 6.2\"\n  DiskPos[1] as DP1\n  DiskPos[2] as DP2\n  DiskPos[3] != 1\n  StepCounter < 1000\n  ogit/RL/totalReward\n  ogit/RL/state\ndo\n  DiskPos[3] = 1\n  ogit/RL/state = \"HanoiPos_${DP1}_${DP2}_1\"\n  ogit/RL/totalReward = ogit/RL/totalReward - 1\n  StepCounter = StepCounter + 1
\n
\n
\n
\n
\n
\n

Single state issue.

\n
\n
\n

Sometimes there is only one state, depending on the issue. In this case the Learning Service will optimize for the number of KIs which were executed during this issue. It means that the Learning Service will try to choose those KIs which will help us to finish the issue faster, with a lower number of timesteps. Another approach is to choose only KIs which produce higher rewards.

\n
\n
\n

A good example of a single state issue is a Multi-armed bandit problem.\nThe state is always the same, but the actions are different (one action for each bandit). The goal is to choose the best actions and get the highest total reward from these actions.

\n
\n
\n
","document":{"main":"Learning Service Guide","title":"Learning Service Guide","subtitle":""},"fields":{"toc":true,"location":["guides","rl-user-guide"]}},"sidebarYaml":{"id":"6d066bdd-c982-5a69-b909-a31e6fc044e0","showIndex":true}},"pageContext":{"id":"f552a231-9b35-5f12-b14f-0863d3bb6782","parent":"guides"}},"staticQueryHashes":["1010459453","1010459453","2356112386","2356112386","2603905930","2603905930","3026652197","3026652197","3167850324","3167850324","63159454","63159454"]}