Event-based Data Capture

Event-based Data Capture refers to the real-time reception of data as it occurs. The specific configuration for enabling this capability may differ across platforms; however, the reported objects will generally contain similar content in both platforms.

Data Objects

These are the different data objects.

SessionUpdate

Property Name Unit Type Description Platform
language name string Current language that the user is typing in. ISO 639-1 - ISO 3166-1 representation. Ex: en-US Both
languageVersion name string Current version for the dictionary language that we are using. Both
layout name string Current layout used for the keyboard, which depend on the language and user preferences. Example: QWERTY, AZERTY, .. Both
textFieldType name enum Current Textfield where the user is writting. Both
appContext name string Current package name of the app where the keyboard is shown. From iOS16 this is not possible, so it might be empty. Both
schemaVersion name string JSON schema version the captured data complies to following semver standard Both
timeZone hours number Current Timezone referenced to GMT Both
startTimestamp ms number Timestamp for session start time Both
endTimestamp ms number Timestamp for session end time. If the sesssion is still active, this value will be zero. Both
keyboardAreaWidth pixels number Keyboard width positioned inside the screen in pixels Both
keyboardAreaHeight pixels number Keyboard height positioned inside the screen in pixels Both
screenWidthMm mm number Screen width of the phone/tablet in mm Both
screenHeightMm mm number Screen height of the phone/tablet in mm Both
screenWidthPx pixels number Screen width of the phone/table in pixels, total number of pixels Both
screenHeightPx pixels number Screen height of the phone/table in pixels, total number of pixels Both
sessionText text string Transcription of the final text input content by the end of the session. Both

TextFieldType

Current Textfield where the user wrote. There might be different values depending on the platform.

This object contains the following values for Android:

  • Text: Regular Text field
  • Password: Password text field
  • URL: URL text field
  • Email_Address: Email text field
  • Numbers: Numbers text field
  • Twitter: Twitter text field, used in Twitter/Slack / …
  • Websearch: Websearch textfield
  • User_AC_OFF: Regular text field without Autocorrection
  • No_Suggestions: Regular text field without suggestions
  • Twitter_AC_OFF: Twitter text field without AC
  • Phone: Phone text field for adding a phone number
  • Undefined: None of the above

This object contains the following values for iOS:

  • regularText: Regular Text field
  • url: URL text field
  • emailAddress: Email text field
  • numbers: Numbers text field
  • twitter: Twitter text field, used in Twitter/Slack / …
  • websearch: Websearch textfield
  • undefined: None of the above

SessionText

sessionText is the accumulated text collected during the session. Value will be filled/available upon session’s termination (last sessionUpdate event within a session).

We have special characters to distinguish different behaviours:

\n When the end-user presses enter we mark this as \n in the reporting
<probable>probable text</probable> We mark text as <probable> when the keyboard is not able to distinguish if that text is present or not. There are two specific cases when this behaviour can be experienced:
1. When the whole text in the textfield is cut.
2. When the whole text in the textfield is selected and the text is committed, e.g. using send button in the Messages app.
This is a new parameter introduced in these versions: iOS 4.14.2 and Android 4.4.21

KeyStroke

The keystroke object is generated when the end-user taps any key.

Property Name Unit Type Description Platform
pressTime ms number Time relative to the startUnixTime reported in the SessionUpdate object. Both
releaseTime ms number Time relative to the startUnixTime reported in the SessionUpdate object. Both
keyPressBegin points (x,y) - Both
keyPressEnd points (x,y) - Both
keyCenter pixels (x,y) Center of the key that we are pressing reference to the screen area in pixels Both
keyRectangle pixels (x,y,width,height) Dimension of the key that we are pressing reference to the screen area in pixels Both
keyType - Key type indicating character family Both
keyCode number number Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32. Both
keyArea - number Description of the area where the user has pressed. Values from 0 to 5 Both
keyText string text Current text pressed, which could be a character, an emoji. Both

keyTypes

Key type indicating character family.

This object contains the following values:

  • alphanumeric: Alphanumeric
  • punctuation: Symbols and punctuation
  • spacebar: Space bar
  • backspace: Backspace
  • enter: Enter
  • modifierKey: Modifier key
  • shift: Shift
  • autoShift: Automatic shift
  • autoToggleNumbers: Automatic numbers
  • toggleEmojis: Emoji button / keyboard from emoji (Note: This value was “em” in the sdk versions iOS 3.X.X and Android 3.X.X)
  • prediction: Prediction from the topbar
  • suggestion: Suggestion from the topbar
  • other: Others

keyCode

Keystroke character equivalent UTF8 Unicode in Decimal. First characters its equivalent to ASCII32.

Note:

  • Number 000 makes reference to actions.
  • Number 900 makes reference to emoji. The transcription of this emoji it is in the array of emojis, by order.
  • Number 901 makes reference to a prediction.

keyArea

Description of the area where the user has pressed. This is a keyArea enum:

This object contains the following values:

  • undefined
  • QWASZ
  • ERDFXC
  • TYGV
  • UIHJBN
  • OPKLM
  • suggestionBar (Note: Versions previous of iOS 4.8.4 and Android 4.2.3 was “9”)

Word

Information related to each of the words.

Property Name Type Description
originalWord string Original word written (in case of AC, Prediction or swipe)
word string Word recorded
isAutoCorrectedWord number It comes from an autocorrection if 1.
isPredictedWord number It comes from a prediction if 1.
isSwipeWord number It comes from a swipe typing if 1.
autocorrectionType enum Type of autocorrection in case of autocorrection true.
autocorrectionValue number Value related to autocorrectionType
timestamp number Time from the start of the session
uniP number Unigram log probability of this word

AutocorrectionType



Values:
missedSpace – in this case value is number of words splitted into
misstypedSpace – Miss an space
eliminated – Characters deleted
transpositioning – Value of characters changed
missingTap – Value of missed taps in the word
none – empty in case of no type

Swipe

Collection of all data related to a trace on top of the keyboard to detect a word.

Property Name Type Description
layout string Specific keyboard layout for swipe. Example: QWERTY, AZERTY, ..
context string Previous word of the swipe
word string Final written word
selectedSuggestion string This is empty unless the user presses one of the suggestions of the topbar.
feature <Array> of <Features> -
candidates <Array> of <Candidates> Array of candidates proposed based on the features.
timestamp number Time from the start of the session.

Features

Property Name Type Description
point (x,y) Point for each of the features
label string Letter detected
timestamp number Path point timestamp from start of the session

Candidates

Property Name Type Description
candidate string Word
shapeProb number Shape log probability
finalProb number Final log probability

Delete

Delete object with the data related to the delete action.

Property Name Type Description
word string Word being delete before applying the delete action
deleteCharacter string Character to delete
typeDelete number Type of delete

deleteCharacter – Delete a Character
deleteSpace – Delete a Space
undoAC – Delete action that undo previous AC
timestamp number Time from the start of the session

KeyPlane

Key plane object is the representation of all the different keys drawn in the screen.

Property Name Object Description
keyPlaneId KeyboardPlaneType
keys Array of Key

KeyboardPlaneType

It holds all different type of keyboard planes:

  • qwertyLower / qwertyUpper / qwertyAC_OffLower / qwertyAC_OffUpper
  • numbers / numberPad / numberPadSymbols / number_iOS / numbersMini
  • symbols / symbols_iOS
  • shift1 / shift2 / shift3
  • phonePad / phonePadSymbols
  • emojis
  • twitterLower / twitterUpper / urlLower / urlUpper / emailLower / emailUpper
  • temp
  • invalid

Key

Property Name Type Description
label string Label associated with key if any
center (x,y) Center of the key
rectangle (x,y,width,height) Rectangle space of the whole key.

StressMonitor

When we have the capability of stress enabled we will be able to receive this object too.

Property Name Type Description
stress Stress Which level of stress is detecting the system.
regime Regime It represents the current status of the system: training or not.

Stress

Property Name Type Description
Status enum Values: stressDetected / stressNotDetected / Undefined

Regime

Property Name Type Description
regime enum Returns if the system is in training mode or `inference mode.

If something needs to be added or if you find an error in our documentation, please let us know either on our GitHub or Discord.

Last updated on May 14, 2024