When the keyboardConfiguration
states that the system is capturing the data per session, you would obtain the following objects at the end of the session.
Data Object
The data captured by the Fleksy Keyboard SDK is in JSON
format for easy parsing and readability and is either:
- Sent as a callback event (Session on Android, DataCollection on iOS), or
- Stored in a file with the
.log
extension at the given location during setup (filename: logger_typing_<timestamp>.log
) as configured.
Example
The below example shows JSON data (object) in a file named logger_typing_b12b8c6a-256a-4fa0-8221-f086f80676b8.log
captured after a session end.
Multiple tags have been collapsed for ease of readability. The captured file contains complete data.
{
"sessionId": "b12b8c6a-256a-4fa0-8221-f086f80676b8",
"startUnixTime": 1673809523412,
"endUnixTime": 1673809546993,
"timeZone": 5.5,
"appContext": "co.thingthing.sample.sdksample",
"schemaVersion": "3.18.0",
"textField": "Text",
"language": "en-US",
"languageVersion": "5100.0",
"layout": "QWERTY",
"screenSizePx": {
"width": 1080.0,
"height": 2400.0
},
"keyboardArea": {
"x": 0.0,
"y": 1739.0,
"width": 1080.0,
"height": 535.0
},
"screenSizeMm": {
"width": 65.314285,
"height": 145.14285
},
"emojis": [],
"wordsData": [
{
"originalWord": "",
"word": "Trying",
"autocorrection": false,
"autocorrectionType": "",
"autocorrectionValue": 0,
"prediction": false,
"swipe": false,
"linkedKeyId": 0,
"timestamp": 14608
}
],
"swipeData": [],
"keyplaneData": {
"switchplanes": [...
],
"keyplanes": [...
]
},
"deleteData": [],
"text": "Trying to type"
}
Initial Setup
Both Android and iOS variants of the Fleksy Keyboard SDK support capturing data which can be configured via the DataCaptureMode
block during the initial keyboard configuration. Furthermore, the data to be captured can also be configured using the FLDataConfiguration
class that is taken as a parameter by the DataCaptureMode
class.
It is recommended to:
- Get explicit consent from the user before capturing data to ensure compliance with required regulations as per regions,
- Store the captured data within user-protected storage on supporting devices as they may contain sensitive information, and
- Enable event logging only in debug or internal builds to avoid leaking the captured information to log files.
Examples
For Android:
To capture data on the Android platform, enable data capture by passing an instance of the CaptureConfiguration class. In addition, the data to be captured can be configured by using the FLDataConfiguration class as dataConfiguration
parameter.
The below code sample showcases configuring data capture by storing the data in the application’s files directory.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
import co.thingthing.fleksy.core.keyboard.KeyboardConfiguration
import co.thingthing.fleksy.core.keyboard.KeyboardConfiguration.CaptureConfiguration
import co.thingthing.fleksy.core.keyboard.KeyboardConfiguration.LicenseConfiguration
import co.thingthing.fleksy.core.keyboard.KeyboardService
import co.thingthing.fleksy.core.keyboard.models.FLDataConfiguration
class SampleKeyboardService : KeyboardService() {
override fun createConfiguration(): KeyboardConfiguration {
return KeyboardConfiguration(
license = LicenseConfiguration(
licenseKey = BuildConfig.FLEKSY_LICENSE_KEY,
licenseSecret = BuildConfig.FLEKSY_SECRET_KEY
),
capture = CaptureConfiguration(
enabled = true,
storeData = true,
location = this.filesDir.absolutePath,
dataConfiguration = FLDataConfiguration()
)
)
}
}
|
For iOS:
Similar to Android, to capture data on the iOS platform, pass an instance of the CaptureConfiguration class. In addition, the captured data can be configured using the dataConfig
parameter during instantiation that takes an instance of the FLDataConfiguration class.
The below code sample showcases configuring data capture by storing the data in the application’s files directory.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
import UIKit
import FleksyKeyboardSDK
class KeyboardViewController: FleksyKeyboardSDK.FKKeyboardViewController {
override func viewDidLoad() {
super.viewDidLoad()
}
override func createConfiguration() -> KeyboardConfiguration {
// CAPTURE DATA
// Capturing by default
// if we want to disable it, pass false instead
let dataConfig = FLDataConfiguration(
configFormat: .dataConfigFormat_groupByTap,
accelerometer: true
)
let capture = CaptureConfiguration(
true,
output: enumCaptureOutput.captureOutput_string,
dataConfig: dataConfig
)
let licenseConfig = LicenseConfiguration(
licenseKey: "<your-license-key>",
licenseSecret: "<your-license-secret>"
)
return KeyboardConfiguration(
capture: capture,
license: licenseConfig
)
}
}
|
Properties
A data object has the following properties:
Property Name |
Unit |
Type |
Period |
Description |
Platform |
startUnixTime |
ms |
<Number> as [Integer] |
Session |
Timestamp for session start time |
Both |
endUnixTime |
- |
- |
Session |
Timestamp for session end time |
Both |
textField |
name |
<String> as [UTF8] |
Session |
Current Textfield where the user wrote. |
Both |
timeZone |
hours |
<Number> as [Float] |
Session |
Current Timezone referenced to GMT |
Both |
language |
name |
<String> as [UTF8] |
Session |
Current language that the user is typing in. en-US |
Both |
layout |
name |
<String> as [UTF8] |
Session |
Current layout used for the keyboard, which depend on the language and user preferences |
Both |
performance |
number |
<Array> of <Numbers> as [Float] |
Event |
Current CPU usage to understand the progression of the device |
iOS |
screenSizeMm |
mm |
{width: <Number> , height: <Number> } as [Floats] |
Session |
Screen size of the phone/tablet in mm |
Both |
screenSizePx |
pixels |
{width: <Number> , height: <Number> } as [Integer] |
Session |
Screen size of the phone/table in pixels, total number of pixels |
Both |
appContext |
name |
<String> as [UTF8] |
Session |
Current package name of the app where the keyboard is shown |
Both |
schemaVersion |
name |
<String> as [UTF8] |
Session |
JSON schema version the captured data complies to following semver standard |
Both |
keyboardArea |
pixels |
{x: <Number>, y: <Number>, width: <Number> , height: <Number> } as [Integer] |
Session |
Keyboard area positioned inside the screen in pixels |
Both |
accelerometerData |
object |
<Array> of <accelerometerObj> |
Event |
(x,y,z,timestamp) position of the accelerometer every x Hz |
Both |
deleteData |
object |
<Array> of <deleteObj> |
Event |
Record all delete events |
Both |
distanceFromLastTouch |
number |
<Array> of <Numbers> as [Float] |
Event |
Euclidean distance from last touch (previous touch end to touch begin) |
Both |
emojis |
string |
<Array> of <String> as [UTF32 as hex number] |
Event |
Transcription of the emojis typed |
Both |
interKeyTimeHistogram |
number |
<Array> of 21 <Numbers> as [Integer] |
Event |
Interkey time histogram (previous touch begin to touch begin) |
Both |
keyCode |
number |
<Array> of <Numbers> as [Integer] |
keystroke |
Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32. |
Both |
keyTypes |
- |
<Array> of <String> as [UTF8] |
keystroke |
Key type indicating character family |
Both |
keyArea |
number |
<Array> of <Numbers> as [Integer] |
keystroke |
Description of the area where the user has pressed. Values from 0 to 6 |
Both |
keyPress |
pixels |
<Array> of <{x,y}> as [Float] |
keystroke |
Position of the keypress reference to the screen area in pixels |
Both |
keyCenter |
pixels |
<Array> of <{x,y}> as [Float] |
keystroke |
Center of the key that we are pressing reference to the screen area in pixels |
Both |
keyBounds |
pixels |
<Array> of <{x,y,width,height}> as [Float] |
keystroke |
Dimension of the key that we are pressing reference to the screen area in pixels |
Both |
keyplaneData |
object |
Object of two elements: switchplanes: <Array> of <switchplanesObj> , keyplanes: <Array> of <keyplanesObj> |
Event |
Record all key plane events change + the center and bounds of each key of the plane |
Both |
swipeData |
object string and numbers |
<Array> of <swipeObj> |
Word |
Collection of all data related to a trace on top of the keyboard to detect a word |
Both |
tapEvent |
- |
- |
- |
- |
- |
text |
- |
<String> as [UTF8] |
Session |
Transcription of the final text input content by end of session |
Both |
wordsData |
object string and numbers |
<Array> of <wordsObj> |
Word |
Information related to each of the words |
Both |
accelerometerData
Position (x,y,z,timestamp) of the accelerometer every x Hz. It contains an <Array> of <accelerometerObj>
.
An instance of accelerometerObj
contains the following values:
Property Name |
Type |
Description |
x |
<number> as [Float] |
x position |
y |
<number> as [Float] |
y position |
z |
<number> as [Float] |
z position |
timestamp |
number |
Time from the start of the session |
deleteData
Record all delete events. It contains an <Array> of <deleteObj>
.
An instance of deleteObj
contains the following values:
Property Name |
Type |
Description |
word |
string |
Word being delete before applying the delete action |
delete |
string |
Character to delete |
type |
number |
Type of delete 0 – Delete a Character 1 – Delete a Space 2 – Delete action that undo previous AC |
timestamp |
number |
Time from the start of the session |
interKeyTimeHistogram
This histogram will be stored as <Array> of 21 <Numbers> as [Integer]
. It indicates the count of interkey times in a typing session that are in each of the bins as specified below:
- The first bin 0-0.05 sec,
- The second 0.05 to 0.10 sec,
- The third 0.10-0.15 sec,
- …,
- The 20th bin 0.95-1.0 sec.
keyArea
Description of the area where the user has pressed. Values from 0 to 6.
This object contains the following values:
- “0” for others
- “1” for letters QWASZ
- “2” for letters ERDFXC
- “3” for letters TYGV
- “4” for letters UIHJBN
- “5” for letters OPKLM
- “6” for Predictions
keyCode
Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32.
Note:
- Number 000 makes reference to actions.
- Number 900 makes reference to emoji. The transcription of this emoji it is in the array of emojis, by order.
- Number 901 makes reference to a prediction.
keyTypes
Key type indicating character family.
This object contains the following values:
- “an”: Alphanumeric
- “pt”: Symbols and punctuation
- “sp”: Space bar
- “bk”: Backspace
- “en”: Enter
- “md”: Modifier key
- “shf”: Shift
- “atcshf”: Automatic shift
- “atcmd”: Automatic numbers
- “eksw”: Emoji button / keyboard from emoji
- “pred”: Prediction from the topbar
- “sugg”: Suggestion from the topbar
- “ukn”: Others
keyplaneData
Record all key plane events change + the center and bounds of each key of the plane. It contains an <Array> of <switchplanesObj>
and <Array> of <keyplanesObj>
.
For example:
{
"keyplaneData": {
"switchplanes": [
{
"keyplane": 0,
"label": "",
"keytype": "atcshf",
"timestamp": 5554
}
],
"keyplanes": [
{
"keyplane": 0,
"keys": [
{
"center": {
"x": 54.0,
"y": 1805.8749980926514
},
"bounds": {
"x": 0.0,
"y": 1739.0,
"width": 108.0,
"height": 133.74999618530273
},
"label": "q"
},
{
"center": {
"x": 162.0,
"y": 1805.8749980926514
},
"bounds": {
"x": 108.0,
"y": 1739.0,
"width": 108.0,
"height": 133.74999618530273
},
"label": "w"
}
]
}
]
},
...
},
}
An instance of switchplanesObj
contains the following values:
Property Name |
Type |
Description |
keyplane |
number |
Type of keyplane – these values are present in the FLEnums.h if you are interested |
label |
<string> as [UTF8] |
Label associated with key if any |
keytype |
<string> as [UTF8] |
Keytype of the key pressed from the list of keytypes. |
timestamp |
number |
Time from the start of the session |
An instance of keyplanesObj
contains the following values:
Property Name |
Type |
Description |
keyplane |
string |
Type of keyplane |
keys |
<Array> of <KeysObj> |
- |
An instance of KeysObj
contains the following values:
Property Name |
Type |
Description |
center |
- |
“x”: <number> as [Float] “y”: <number> as [Float] |
bounds |
- |
“x”: <number> as [Float] “y”: <number> as [Float] “width”: <number> as [Float] “height”: <number> as [Float] |
label |
<string> as [UTF8] |
Letter detected |
swipeData
Collection of all data related to a trace on top of the keyboard to detect a word. It contains an <Array> of <swipeObj>
.
An instance of swipeObj
contains the following values:
Property Name |
Type |
Description |
layout |
string |
Word being delete before applying the delete action |
context |
string |
Previous word of the swipe |
candidate |
string |
Word that we’re guessing |
word |
string |
Word final written |
selectedSuggestion |
string |
This is empty unless the user presses one of the suggestions of the topbar. |
points |
<Array> of <Points> |
- |
timestamp |
number |
Time from the start of the session |
candidates |
<Array> of <Candidates> |
- |
An instance of Points
contains the following values:
Property Name |
Type |
Description |
x |
<number> as [Float] |
x position |
y |
<number> as [Float] |
y position |
l |
<string> as [UTF8] |
Letter detected |
t |
<number> |
Path point timestamp from start of the session |
An instance of Candidates
contains the following values:
Property Name |
Type |
Description |
w |
<string> as [UTF8] |
Word |
s |
<number> |
Shape log probability |
f |
<number> |
Final log probability |
tapEvent
The tapEvent object gets generated in case of selecting the option of “Format group by tap”, which groups all keyStroks in one object per key stroke.
Property Name |
Unit |
Type |
Period |
Description |
Platform |
keyArea |
number |
<Array> of <Numbers> as [Integer] |
keystroke |
Description of the area where the user has pressed. Values from 0 to 6 |
Both |
keyBounds |
pixels |
<Array> of <{x,y,width,height}> as [Float] |
keystroke |
Dimension of the key that we are pressing reference to the screen area in pixels |
Both |
keyCenter |
pixels |
<Array> of <{x,y}> as [Float] |
keystroke |
Center of the key that we are pressing reference to the screen area in pixels |
Both |
keyCode |
number |
<Array> of <Numbers> as [Integer] |
keystroke |
Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32. |
Both |
keyIdentifier |
- |
- |
- |
Different <number> for each tapEvent during a session. It restarts every session. |
Both |
keyText |
- |
- |
- |
Current text pressed, which could be a character, an emoji, prediction word |
Both |
keyTypes |
- |
<Array> of <String> as [UTF8] |
keystroke |
Key type indicating character family |
Both |
touchBegin |
- |
- |
- |
- |
- |
touchEnd |
- |
- |
- |
- |
- |
The properties touchBegin
and touchEnd
contains the following values:
- “press”:
<obj>
| position of the press
- “x”:
<number>
| x position
- “y”:
<number>
| y position
- “distance”:
<obj>
| distance from between the center of the key and the press position
- “x”:
<number>
| x position
- “y”:
<number>
| y position
- “timestamp”:
<number>
| time from the start of the session
textField
Current Textfield where the user wrote.
This object contains the following values:
- “Text”: Regular Text field
- “Pwd”: Password text field
- “Url”: URL text field
- “Email”: Email text field
- “Numbers”: Numbers text field
- “Twitter”: Twitter text field, used in Twitter/Slack / …
- “Websearch”: Websearch textfield
- “Text_ACOFF”: Regular text field without Autocorrection
- “Twitter_ACOFF”: Twitter text field without AC
text
text
represents the accumulative text of the session.
The text appears as the end-user sends it.
We have special characters to distinguish different behaviours:
|
|
\n |
When the end-user presses enter we mark this as \n in the reporting |
<probable>probable text</probable> |
We mark text as <probable> when the keyboard is not able to distinguish if that text is present or not. There are two specific cases when this behaviour can be experienced: 1. When the whole text in the textfield is cut . 2. When the whole text in the textfield is selected and the text is committed , e.g. using send button in the Messages app. This is a new parameter introduced in these versions: iOS 4.14.2 and Android 4.4.21 |
wordsData
Information related to each of the words. It contains an <Array> of <wordsObj>
.
For example:
{
"wordsData": [
{
"originalWord": "",
"word": "Trying",
"autocorrection": false,
"autocorrectionType": "",
"autocorrectionValue": 0,
"prediction": false,
"swipe": false,
"linkedKeyId": 0,
"timestamp": 14608
}
],
...
},
}
An instance of wordsObj
contains the following values:
Property Name |
Type |
Description |
originalWord |
string |
Original word written (in case of AC, Prediction or swipe) |
word |
string |
Word recorded |
autocorrection |
bool |
It comes from an autocorrection |
prediction |
bool |
It comes from a prediction |
swipe |
bool |
It comes from a swipe typing |
autocorrectionType |
string |
Type of autocorrection in case of autocorrection true.Values:“missedSpace” – in this case value is number of words splitted into “missTypedSpace” – Miss an space “eliminated” – Characters deleted “transposition” – Value of characters changed “missingTap” – Value of missed taps in the word ”" – empty in case of no type |
autocorrectionValue |
number |
Value related to autocorrectionType |
timestamp |
number |
Time from the start of the session |
linkedKeyId |
number |
Linked id with the tapEvent that caused this word. When the word is autocorrected, it would be associated with the corresponding Id in the tapEvent array. “0” means no id linked from tapEvent. |
unip |
number |
Unigram log probability of this word |