Session-based Data Capture

When the keyboardConfiguration states that the system is capturing the data per session, you would obtain the following objects at the end of the session.

Data Object

The data captured by the Fleksy Keyboard SDK is in JSON format for easy parsing and readability and is either:

  • Sent as a callback event (Session on Android, DataCollection on iOS), or
  • Stored in a file with the .log extension at the given location during setup (filename: logger_typing_<timestamp>.log) as configured.

Example

The below example shows JSON data (object) in a file named logger_typing_b12b8c6a-256a-4fa0-8221-f086f80676b8.log captured after a session end.

Multiple tags have been collapsed for ease of readability. The captured file contains complete data.

{
    "sessionId": "b12b8c6a-256a-4fa0-8221-f086f80676b8",
    "startUnixTime": 1673809523412,
    "endUnixTime": 1673809546993,
    "timeZone": 5.5,
    "appContext": "co.thingthing.sample.sdksample",
    "schemaVersion": "3.18.0",
    "textField": "Text",
    "language": "en-US",
    "languageVersion": "5100.0",
    "layout": "QWERTY",
    "screenSizePx": {
        "width": 1080.0,
        "height": 2400.0
    },
    "keyboardArea": {
        "x": 0.0,
        "y": 1739.0,
        "width": 1080.0,
        "height": 535.0
    },
    "screenSizeMm": {
        "width": 65.314285,
        "height": 145.14285
    },
    "emojis": [],
    "wordsData": [
        {
            "originalWord": "",
            "word": "Trying",
            "autocorrection": false,
            "autocorrectionType": "",
            "autocorrectionValue": 0,
            "prediction": false,
            "swipe": false,
            "linkedKeyId": 0,
            "timestamp": 14608
        }
    ],
    "swipeData": [],
    "keyplaneData": {
        "switchplanes": [...
        ],
        "keyplanes": [...
        ]
    },
    "deleteData": [],
    "text": "Trying to type"
}

Initial Setup

Both Android and iOS variants of the Fleksy Keyboard SDK support capturing data which can be configured via the DataCaptureMode block during the initial keyboard configuration. Furthermore, the data to be captured can also be configured using the FLDataConfiguration class that is taken as a parameter by the DataCaptureMode class.

It is recommended to:

  • Get explicit consent from the user before capturing data to ensure compliance with required regulations as per regions,
  • Store the captured data within user-protected storage on supporting devices as they may contain sensitive information, and
  • Enable event logging only in debug or internal builds to avoid leaking the captured information to log files.

Examples

For Android:

To capture data on the Android platform, enable data capture by passing an instance of the CaptureConfiguration class. In addition, the data to be captured can be configured by using the FLDataConfiguration class as dataConfiguration parameter.

The below code sample showcases configuring data capture by storing the data in the application’s files directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import co.thingthing.fleksy.core.keyboard.KeyboardConfiguration
import co.thingthing.fleksy.core.keyboard.KeyboardConfiguration.CaptureConfiguration
import co.thingthing.fleksy.core.keyboard.KeyboardConfiguration.LicenseConfiguration
import co.thingthing.fleksy.core.keyboard.KeyboardService
import co.thingthing.fleksy.core.keyboard.models.FLDataConfiguration

class SampleKeyboardService : KeyboardService() {

    override fun createConfiguration(): KeyboardConfiguration {
        return KeyboardConfiguration(
            license = LicenseConfiguration(
                licenseKey = BuildConfig.FLEKSY_LICENSE_KEY,
                licenseSecret = BuildConfig.FLEKSY_SECRET_KEY
            ),
            capture = CaptureConfiguration(
                enabled = true,
                storeData = true,
                location = this.filesDir.absolutePath,
                dataConfiguration = FLDataConfiguration()
            )
        )
    }

}

For iOS:

Similar to Android, to capture data on the iOS platform, pass an instance of the CaptureConfiguration class. In addition, the captured data can be configured using the dataConfig parameter during instantiation that takes an instance of the FLDataConfiguration class.

The below code sample showcases configuring data capture by storing the data in the application’s files directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import UIKit
import FleksyKeyboardSDK

class KeyboardViewController: FleksyKeyboardSDK.FKKeyboardViewController {

    override func viewDidLoad() {
        super.viewDidLoad()
    }

    override func createConfiguration() -> KeyboardConfiguration {

        // CAPTURE DATA
        // Capturing by default
        // if we want to disable it, pass false instead
        let dataConfig = FLDataConfiguration(
            configFormat: .dataConfigFormat_groupByTap,
            accelerometer: true
        )
        let capture = CaptureConfiguration(
            true,
            output: enumCaptureOutput.captureOutput_string,
            dataConfig: dataConfig
        )

        let licenseConfig = LicenseConfiguration(
            licenseKey: "<your-license-key>",
            licenseSecret: "<your-license-secret>"
        )

        return KeyboardConfiguration(
            capture: capture,
            license: licenseConfig
        )
    }
}

Properties

A data object has the following properties:

Property Name Unit Type Period Description Platform
startUnixTime ms <Number> as [Integer] Session Timestamp for session start time Both
endUnixTime - - Session Timestamp for session end time Both
textField name <String> as [UTF8] Session Current Textfield where the user wrote. Both
timeZone hours <Number> as [Float] Session Current Timezone referenced to GMT Both
language name <String> as [UTF8] Session Current language that the user is typing in. en-US Both
layout name <String> as [UTF8] Session Current layout used for the keyboard, which depend on the language and user preferences Both
performance number <Array> of <Numbers> as [Float] Event Current CPU usage to understand the progression of the device iOS
screenSizeMm mm {width: <Number> , height: <Number> } as [Floats] Session Screen size of the phone/tablet in mm Both
screenSizePx pixels {width: <Number> , height: <Number> } as [Integer] Session Screen size of the phone/table in pixels, total number of pixels Both
appContext name <String> as [UTF8] Session Current package name of the app where the keyboard is shown Both
schemaVersion name <String> as [UTF8] Session JSON schema version the captured data complies to following semver standard Both
keyboardArea pixels {x: <Number>, y: <Number>, width: <Number> , height: <Number> } as [Integer] Session Keyboard area positioned inside the screen in pixels Both
accelerometerData object <Array> of <accelerometerObj> Event (x,y,z,timestamp) position of the accelerometer every x Hz Both
deleteData object <Array> of <deleteObj> Event Record all delete events Both
distanceFromLastTouch number <Array> of <Numbers> as [Float] Event Euclidean distance from last touch (previous touch end to touch begin) Both
emojis string <Array> of <String> as [UTF32 as hex number] Event Transcription of the emojis typed Both
interKeyTimeHistogram number <Array> of 21 <Numbers> as [Integer] Event Interkey time histogram (previous touch begin to touch begin) Both
keyCode number <Array> of <Numbers> as [Integer] keystroke Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32. Both
keyTypes - <Array> of <String> as [UTF8] keystroke Key type indicating character family Both
keyArea number <Array> of <Numbers> as [Integer] keystroke Description of the area where the user has pressed. Values from 0 to 6 Both
keyPress pixels <Array> of <{x,y}> as [Float] keystroke Position of the keypress reference to the screen area in pixels Both
keyCenter pixels <Array> of <{x,y}> as [Float] keystroke Center of the key that we are pressing reference to the screen area in pixels Both
keyBounds pixels <Array> of <{x,y,width,height}> as [Float] keystroke Dimension of the key that we are pressing reference to the screen area in pixels Both
keyplaneData object Object of two elements: switchplanes: <Array> of <switchplanesObj>, keyplanes: <Array> of <keyplanesObj> Event Record all key plane events change + the center and bounds of each key of the plane Both
swipeData object string and numbers <Array> of <swipeObj> Word Collection of all data related to a trace on top of the keyboard to detect a word Both
tapEvent - - - - -
text - <String> as [UTF8] Session Transcription of the final text input content by end of session Both
wordsData object string and numbers <Array> of <wordsObj> Word Information related to each of the words Both

accelerometerData

Position (x,y,z,timestamp) of the accelerometer every x Hz. It contains an <Array> of <accelerometerObj>.

An instance of accelerometerObj contains the following values:

Property Name Type Description
x <number> as [Float] x position
y <number> as [Float] y position
z <number> as [Float] z position
timestamp number Time from the start of the session

deleteData

Record all delete events. It contains an <Array> of <deleteObj>.

An instance of deleteObj contains the following values:

Property Name Type Description
word string Word being delete before applying the delete action
delete string Character to delete
type number Type of delete

0 – Delete a Character
1 – Delete a Space
2 – Delete action that undo previous AC
timestamp number Time from the start of the session

interKeyTimeHistogram

This histogram will be stored as <Array> of 21 <Numbers> as [Integer]. It indicates the count of interkey times in a typing session that are in each of the bins as specified below:

  • The first bin 0-0.05 sec,
  • The second 0.05 to 0.10 sec,
  • The third 0.10-0.15 sec,
  • …,
  • The 20th bin 0.95-1.0 sec.

keyArea

Description of the area where the user has pressed. Values from 0 to 6.

This object contains the following values:

  • “0” for others
  • “1” for letters QWASZ
  • “2” for letters ERDFXC
  • “3” for letters TYGV
  • “4” for letters UIHJBN
  • “5” for letters OPKLM
  • “6” for Predictions

keyCode

Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32.

Note:

  • Number 000 makes reference to actions.
  • Number 900 makes reference to emoji. The transcription of this emoji it is in the array of emojis, by order.
  • Number 901 makes reference to a prediction.

keyTypes

Key type indicating character family.

This object contains the following values:

  • “an”: Alphanumeric
  • “pt”: Symbols and punctuation
  • “sp”: Space bar
  • “bk”: Backspace
  • “en”: Enter
  • “md”: Modifier key
  • “shf”: Shift
  • “atcshf”: Automatic shift
  • “atcmd”: Automatic numbers
  • “eksw”: Emoji button / keyboard from emoji
  • “pred”: Prediction from the topbar
  • “sugg”: Suggestion from the topbar
  • “ukn”: Others

keyplaneData

Record all key plane events change + the center and bounds of each key of the plane. It contains an <Array> of <switchplanesObj> and <Array> of <keyplanesObj>.

For example:

{
    "keyplaneData": {
        "switchplanes": [
            {
                "keyplane": 0,
                "label": "",
                "keytype": "atcshf",
                "timestamp": 5554
            }
        ],
        "keyplanes": [
            {
                "keyplane": 0,
                "keys": [
                    {
                        "center": {
                            "x": 54.0,
                            "y": 1805.8749980926514
                        },
                        "bounds": {
                            "x": 0.0,
                            "y": 1739.0,
                            "width": 108.0,
                            "height": 133.74999618530273
                        },
                        "label": "q"
                    },
                    {
                        "center": {
                            "x": 162.0,
                            "y": 1805.8749980926514
                        },
                        "bounds": {
                            "x": 108.0,
                            "y": 1739.0,
                            "width": 108.0,
                            "height": 133.74999618530273
                        },
                        "label": "w"
                    }
                ]
            }
        ]
    },
    ...
  },
}

An instance of switchplanesObj contains the following values:

Property Name Type Description
keyplane number Type of keyplane – these values are present in the FLEnums.h if you are interested
label <string> as [UTF8] Label associated with key if any
keytype <string> as [UTF8] Keytype of the key pressed from the list of keytypes.
timestamp number Time from the start of the session

An instance of keyplanesObj contains the following values:

Property Name Type Description
keyplane string Type of keyplane
keys <Array> of <KeysObj> -

An instance of KeysObj contains the following values:

Property Name Type Description
center - “x”: <number> as [Float]
“y”: <number> as [Float]
bounds - “x”: <number> as [Float]
“y”: <number> as [Float]
“width”: <number> as [Float]
“height”: <number> as [Float]
label <string> as [UTF8] Letter detected

swipeData

Collection of all data related to a trace on top of the keyboard to detect a word. It contains an <Array> of <swipeObj>.

An instance of swipeObj contains the following values:

Property Name Type Description
layout string Word being delete before applying the delete action
context string Previous word of the swipe
candidate string Word that we’re guessing
word string Word final written
selectedSuggestion string This is empty unless the user presses one of the suggestions of the topbar.
points <Array> of <Points> -
timestamp number Time from the start of the session
candidates <Array> of <Candidates> -

An instance of Points contains the following values:

Property Name Type Description
x <number> as [Float] x position
y <number> as [Float] y position
l <string> as [UTF8] Letter detected
t <number> Path point timestamp from start of the session

An instance of Candidates contains the following values:

Property Name Type Description
w <string> as [UTF8] Word
s <number> Shape log probability
f <number> Final log probability

tapEvent

The tapEvent object gets generated in case of selecting the option of “Format group by tap”, which groups all keyStroks in one object per key stroke.

Property Name Unit Type Period Description Platform
keyArea number <Array> of <Numbers> as [Integer] keystroke Description of the area where the user has pressed. Values from 0 to 6 Both
keyBounds pixels <Array> of <{x,y,width,height}> as [Float] keystroke Dimension of the key that we are pressing reference to the screen area in pixels Both
keyCenter pixels <Array> of <{x,y}> as [Float] keystroke Center of the key that we are pressing reference to the screen area in pixels Both
keyCode number <Array> of <Numbers> as [Integer] keystroke Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32. Both
keyIdentifier - - - Different <number> for each tapEvent during a session. It restarts every session. Both
keyText - - - Current text pressed, which could be a character, an emoji, prediction word Both
keyTypes - <Array> of <String> as [UTF8] keystroke Key type indicating character family Both
touchBegin - - - - -
touchEnd - - - - -

The properties touchBegin and touchEnd contains the following values:

  • “press”: <obj> | position of the press
    • “x”: <number> | x position
    • “y”: <number> | y position
  • “distance”: <obj> | distance from between the center of the key and the press position
    • “x”: <number> | x position
    • “y”: <number> | y position
  • “timestamp”: <number> | time from the start of the session

textField

Current Textfield where the user wrote.

This object contains the following values:

  • “Text”: Regular Text field
  • “Pwd”: Password text field
  • “Url”: URL text field
  • “Email”: Email text field
  • “Numbers”: Numbers text field
  • “Twitter”: Twitter text field, used in Twitter/Slack / …
  • “Websearch”: Websearch textfield
  • “Text_ACOFF”: Regular text field without Autocorrection
  • “Twitter_ACOFF”: Twitter text field without AC

text

text represents the accumulative text of the session. The text appears as the end-user sends it.

We have special characters to distinguish different behaviours:

\n When the end-user presses enter we mark this as \n in the reporting
<probable>probable text</probable> We mark text as <probable> when the keyboard is not able to distinguish if that text is present or not. There are two specific cases when this behaviour can be experienced:
1. When the whole text in the textfield is cut.
2. When the whole text in the textfield is selected and the text is committed, e.g. using send button in the Messages app.
This is a new parameter introduced in these versions: iOS 4.14.2 and Android 4.4.21

wordsData

Information related to each of the words. It contains an <Array> of <wordsObj>.

For example:

{
    "wordsData": [
        {
            "originalWord": "",
            "word": "Trying",
            "autocorrection": false,
            "autocorrectionType": "",
            "autocorrectionValue": 0,
            "prediction": false,
            "swipe": false,
            "linkedKeyId": 0,
            "timestamp": 14608
        }
    ],
    ...
  },
}

An instance of wordsObj contains the following values:

Property Name Type Description
originalWord string Original word written (in case of AC, Prediction or swipe)
word string Word recorded
autocorrection bool It comes from an autocorrection
prediction bool It comes from a prediction
swipe bool It comes from a swipe typing
autocorrectionType string Type of autocorrection in case of autocorrection true.

Values:
“missedSpace” – in this case value is number of words splitted into
“missTypedSpace” – Miss an space
“eliminated” – Characters deleted
“transposition” – Value of characters changed
“missingTap” – Value of missed taps in the word
”" – empty in case of no type
autocorrectionValue number Value related to autocorrectionType
timestamp number Time from the start of the session
linkedKeyId number Linked id with the tapEvent that caused this word. When the word is autocorrected, it would be associated with the corresponding Id in the tapEvent array. “0” means no id linked from tapEvent.
unip number Unigram log probability of this word

If something needs to be added or if you find an error in our documentation, please let us know either on our GitHub or Discord.

Last updated on May 14, 2024