Retreiving text id (`"#n": "g-x"`) at runtime for localisation

[GieziJo]

First off, just wanted to say thanks a lot for this amazing tool, we have been using it in our upcoming game Hermit, and it is just awesome to handle dialogues!

The one thing I am a big concerned about is localisation. After reading a lot of opinions I came to the conclusion that avoiding inline variables, using ids for each line and pairing this with a dictionary would be the way to go (here, here and inkle/ink#529).
I am not entirely convinced by the solutions provided in the linked articles to create these IDs though.

Looking at the generated .json, it seems like every text line already contains a tag, something of the sorts of "#n": "g-3". Pair this with the current knot and this creates a unique ID.

My question is thus:

Is it possible to retrieve this g-tag at runtime?

I'm thinking about something of the like of story.currentID (which does not exists).
I wouldn't mind adding this to the integration, but not sure where to do it.
In JsonSerialisation.cs I saw the lines

if (keyVal.Key == "#n") 
   container.name = keyVal.Value.ToString ();

But I haven't been able to figure out how the tag is then used.

Extracting all the lines from the json should be easy enough if that part is possible.

Thanks!

[tomkail]

This is an interesting idea! I don't have an answer to your question but if you or anyone else finds a solution please share it here, it seems like a problem worth solving :)

[GieziJo]

thanks!
Gonna try to look into it, but I've had a hard time understanding the structure of containers so far 😅

[GieziJo]

Ok, think I found a pretty good solution (should at least work for my use case):

most of the path can be used as a key, works pretty well. You just need to remove the index at the end, unused to generate the key:

story.Continue ();
string key = story.state.currentPointer.path.ToString()[..^2];

To generate the dictionary to be exported as csv, this seems to be working:

private static Dictionary<string, string> dialogueDictionary;

public static void BuildDictionary()
{
    string jsonFile = AssetDatabase.LoadAssetAtPath<TextAsset>("Assets/Editor/Localisation/test.json").ToString();
    Dictionary<string, object> rootObject = SimpleJson.TextToDictionary(jsonFile);
    
    var rootToken = rootObject ["root"];

    Container container = Json.JTokenToRuntimeObject (rootToken) as Container;

    dialogueDictionary = new Dictionary<string, string>();
    
    foreach (KeyValuePair<string,Object> keyValuePair in container.namedOnlyContent)
    {
        if(keyValuePair.Value is Container)
            ExtractContent((Container)keyValuePair.Value);
    }
    
}

static void ExtractContent(Container container)
{
    if(container.content != null)
        foreach (Object obj in container.content)
        {
            if(obj is Ink.Runtime.StringValue && obj.ToString().Trim().Length !=0)
            {
                dialogueDictionary.Add(container.path.ToString(), obj.ToString());
            }
            if(obj is Container)
                ExtractContent((Container)obj);
        }
}

(obviously replace Assets/Editor/Localisation/test.json with a path to your file).

This can then easily be exported as csv for localisation. Still implementing the details, but seems to work so far :D

[tomkail]

Awesome!! Please keep us posted on how it goes, I’d love to try this out myself.

…

On Sat, 21 May 2022 at 15:08, Giezi ***@***.***> wrote: Ok, think I fund a pretty good solution (should at least work for my use case): most of the path can be used as a key, works pretty well. You just need to remove the index at the end, unused to generate the key: story.Continue ();string key = story.state.currentPointer.path.ToString();key = key.Substring(0, key.Length - 2); To generate the dictionary to be exported as csv, this seems to be working: private static Dictionary<string, string> dialogueDictionary; public static void BuildDictionary() { string jsonFile = AssetDatabase.LoadAssetAtPath<TextAsset>("Assets/Editor/Localisation/test.json").ToString(); Dictionary<string, object> rootObject = SimpleJson.TextToDictionary(jsonFile); var rootToken = rootObject ["root"]; Container container = Json.JTokenToRuntimeObject (rootToken) as Container; dialogueDictionary = new Dictionary<string, string>(); foreach (KeyValuePair<string,Object> keyValuePair in container.namedOnlyContent) { if(keyValuePair.Value is Container) ExtractContent((Container)keyValuePair.Value); } } static void ExtractContent(Container container) { if(container.content != null) foreach (Object obj in container.content) { if(obj is Ink.Runtime.StringValue && obj.ToString().Trim().Length !=0) { dialogueDictionary.Add(container.path.ToString(), obj.ToString()); } if(obj is Container) ExtractContent((Container)obj); } } (obviously replace Assets/Editor/Localisation/test.json with a path to your file). This can then easily be exported as csv for localisation. Still implementing the details, but seems to work so far :D — Reply to this email directly, view it on GitHub <#166 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJR3UEGYIMG2UPLYZTFX5LVLDU45ANCNFSM5VQITKJA> . You are receiving this because you commented.Message ID: ***@***.***>

[tomkail]

Hi @GieziJo! Did this work out for you? If it did, would you be willing to share it with the community? This seems like something others might find handy!

[GieziJo]

Hi Sorry I completely missed your message.

I "kinda" solved it yes, but it's really not perfect, and for now I'm not really using it.
The main reason I'm not using it, is it will fail if anything is changed in the ink files, because then the key changes.
I would need a way to update the key dynamically in my translation spreadsheet, but I haven't gotten to write that part yet.
Right now, what I do, is I just check if the content of the key has changed and flag it if it has.

---

TLDR:

wow this has gotten pretty long, maybe this to summarize.
Extracting a key (note that this is recursive):

void BuildDictionary(string jsonFilePath){

	private static Dictionary<string, string> dialogueDictionary = new Dictionary<string, string>();

	string jsonContent = AssetDatabase.LoadAssetAtPath<TextAsset>(jsonFilePath)
		.ToString();
	object rootObject = SimpleJson.TextToDictionary(jsonContent)["root"];

	Container container = Json.JTokenToRuntimeObject(rootObject) as Container;

	foreach (KeyValuePair<string, Object> keyValuePair in container.namedOnlyContent)
	{
		if (keyValuePair.Value is Container)
			ExtractContent((Container) keyValuePair.Value);
	}
	return dialogueDictionary
}

private static bool _isHashtag = false;

private static void ExtractContent(Container container)
{
	if (container.content != null)
		foreach (Object obj in container.content)
		{
			if (obj.ToString() == "BeginTag")
				_isHashtag = true;
			else if (obj.ToString() == "EndTag")
				_isHashtag = false;
			
			if (!_isHashtag && obj is Ink.Runtime.StringValue && obj.ToString().Trim().Length != 0)
			{
				dialogueDictionary.Add(container.path.ToString(), obj.ToString().Trim());
			}

			if (obj is Container)
				ExtractContent((Container) obj);
		}
}

Getting the key in game (needs more for inline variables, read below if you need that):

string key = story.state.outputStream[0].path.ToString()[..^2];

TLDR Over, read on if you want to know the whole thing

This is my approach:

I first have a script that exports each line and checks for existing content and flags it if so.
This prints the result to the console (could also go to a csv). I work with google docs, so ideally this would update that spreadsheet, but again, haven't gotten to it yet 😅

using System.Collections.Generic;
using System.Linq;
using Ink.Runtime;
using UnityEditor;
using UnityEngine;
using Object = Ink.Runtime.Object;

namespace Localisation.Editor
{
    public static class ExportInkStory
    {

        private static Dictionary<string, string> dialogueDictionary;

        [MenuItem("Utility/Localisation/Export Ink Story")]
        public static void ExportToSheet()
        {
            BuildDictionary();
        }
        
        private static void BuildDictionary()
        {
            dialogueDictionary = new Dictionary<string, string>();
            
            foreach (string jsonFilePath in AssetDatabase.FindAssets("", new[] {"Assets/Dialogues/DialoguesCompiled"})
                         .Select(AssetDatabase.GUIDToAssetPath).Where(s => s.Contains("json")).ToList())
            {
                Debug.Log(jsonFilePath);
                string jsonContent = AssetDatabase.LoadAssetAtPath<TextAsset>(jsonFilePath)
                    .ToString();
                object rootObject = SimpleJson.TextToDictionary(jsonContent)["root"];

                Container container = Json.JTokenToRuntimeObject(rootObject) as Container;

                foreach (KeyValuePair<string, Object> keyValuePair in container.namedOnlyContent)
                {
                    if (keyValuePair.Value is Container)
                        ExtractContent((Container) keyValuePair.Value);
                }
            }
        }

        [MenuItem("Utility/Localisation/Print Ink Story as CSV")]
        public static void PrintToConsole() => Debug.Log(BuildString());
        private static string BuildString()
        {
            BuildDictionary();
            string outString = "";
            foreach (KeyValuePair<string, string> keyValuePair in dialogueDictionary)
            {
                int hasContentChanged = InkLocalisationDictionary.Instance.GetString(keyValuePair.Key, LocalisationLanguage.En) == keyValuePair.Value ? 0 : 1;
                outString += $"\"{keyValuePair.Key}\", \"{hasContentChanged}\", \"{keyValuePair.Value}\"\n";
            }

            return outString;
        }

	private static bool _isHashtag = false;

	private static void ExtractContent(Container container)
	{
		if (container.content != null)
			foreach (Object obj in container.content)
			{
				if (obj.ToString() == "BeginTag")
					_isHashtag = true;
				else if (obj.ToString() == "EndTag")
					_isHashtag = false;
				
				if (!_isHashtag && obj is Ink.Runtime.StringValue && obj.ToString().Trim().Length != 0)
				{
					dialogueDictionary.Add(container.path.ToString(), obj.ToString().Trim());
				}

				if (obj is Container)
					ExtractContent((Container) obj);
			}
	}
    }
}

The output, once pasted in google sheets, looks like this:

As you can see, this generates a key for each line as this:

CrabBoss_SinglePlayer_Beaten.0.g-0.g-1.g-2.g-3.g-4.g-5.g-6.g-7.g-8.g-9

Here CrabBoss, SinglePlayer and Beaten are tunnels, and 0.g-0.g-1.g-2.g-3.g-4.g-5.g-6.g-7.g-8.g-9 is the line specific key.

This is all generated between BuildDictionary and ExtractContent.

If you don't care about checking for changes, you could ignore int hasContentChanged = InkLocalisationDictionary.Instance.GetString(keyValuePair.Key, LocalisationLanguage.En) == keyValuePair.Value ? 0 : 1; and remove the part that load the previous localisation file.

This does not work with inline variables directly (for example to replace names, or amounts).
I had an idea how to make this work with variables directly (it's been a while, I would have to dig into it), but for now I use a workaround.
For now I use a placeholder.
For example, we have a currency called plankton in the game, when using the variable name, I don't set it in ink, I set it in unity (ink line):

- So fancy an upgrade? The value over there shows you how much plankton you collected, you currently have \{plankton\}. Obviously nothing is free, gotta keep Meredith fed!

Inkle doesn't interpret \{plankton\} as a variable, and thus you can just replace that in unity with string swap.

Once the csv is built, I have a dictionary with the keys and the lines for the different languages (happy to share this also if of interest).

In unity then, to read the lines, I have to lookup the keys.

private ReadStory(){
	while (story.canContinue)
	{
		string text = GetNextText();
	}
}

private string GetNextText()
{
	string text = story.Continue();
	
	string key = story.state.outputStream[0].path.ToString()[..^2];
		
	string transletedText = LocalisationHandler.Instance.GetString(key); // this is where I access my dictionary with the translation

	if (transletedText.Length != 0)
		text = transletedText;
	
	return InsertVariables(text);
}

This works, but what we are missing is the variables (in return InsertVariables(text);).

This is where it gets a little complicated, this is how I did it, but I think there should be a better way.

First, find a replace the variables tagged with \{varname\}:

private string InsertVariables(string text)
{
	var reg = Regex.Matches(text, @"\{([^{}]+)\}");
	
	
	if (reg.Count == 0)
		return text;
	
	List<string> variableNames = reg.Cast<Match>()
		.Select(m => m.Groups[1].Value)
		.Distinct()
		.ToList();

	variableNames = LocalisationVariables.GetValuesForVariable(variableNames);
	
	List<string> original = reg.Cast<Match>()
		.Select(m => m.Groups[0].Value)
		.Distinct()
		.ToList();

	for (int i = 0; i < original.Count; i++)
	{
		text = text.Replace(original[i], variableNames[i]);
	}

	return text;
}

This relies on LocalisationVariables.GetValuesForVariable(variableNames);, which is where we set the variable names and keep them:

public class LocalisationVariables
{
	private static Dictionary<string, Func<string>> _localisationVariables = null;

	private static LocalisationVariablesReferences _localisationVariablesReferences;

	public static string GetValueForVariable(string variable)
	{
		return (_localisationVariables ?? BuildDictionary()).TryGetValue(variable, out Func<string> variableCaller)
			? variableCaller()
			: "";

	}

	static Dictionary<string, Func<string>> BuildDictionary()
	{
		_localisationVariablesReferences = Resources.Load<LocalisationVariablesReferences>("Localisation/LocalisationVariablesReferences");
		_localisationVariables = new Dictionary<string, Func<string>>();
		_localisationVariables.Add("plankton", () => _localisationVariablesReferences.SharedPlayerResources.ShellCardPoints.ToString());
		return _localisationVariables;
	}

	public static List<string> GetValuesForVariable(List<string> variables)
	{
		List<string> values = new List<string>();
		foreach (string variable in variables)
		{
			values.Add(GetValueForVariable(variable));
		}

		return values;
	}
}

Each variable that will be called in the text has a reference to scriptable object, as in

_localisationVariables.Add("plankton", () => _localisationVariablesReferences.SharedPlayerResources.ShellCardPoints.ToString());

_localisationVariablesReferences contains the references to the scriptable objects I need for my variables.

This could probably also just be updated on the fly and simplified, it just made sense in my context.

Sorry this is quite a lot, and it's not perfect, I'd be happy to work on a better solution, but for now it works for our needs.
I wanted to create a package or something to have this as a ready made solution, but I run out of time and don't feel like it's quite there yet.

I'd be happy to work with you all on something if you feel like it 😄

Let me know if something is not clear, which it probably is 😅

[GieziJo]

Playing around with it yesterday I realized it doesn't work on tags anymore, did maybe something change @tomkail ?

I modified the function a bit and this works, but maybe there is a more elegant solution?
(editing the function in the post above too)

private static bool _isHashtag = false;

private static void ExtractContent(Container container)
{
	if (container.content != null)
		foreach (Object obj in container.content)
		{
			if (obj.ToString() == "BeginTag")
				_isHashtag = true;
			else if (obj.ToString() == "EndTag")
				_isHashtag = false;
			
			if (!_isHashtag && obj is Ink.Runtime.StringValue && obj.ToString().Trim().Length != 0)
			{
				dialogueDictionary.Add(container.path.ToString(), obj.ToString().Trim());
			}

			if (obj is Container)
				ExtractContent((Container) obj);
		}
}

I also started playing around with the Jaro Winkler Distance yesterday when updating the csv, to see if any of the old strings match, and then copy the translation for the highest score, and it looks like it's working fairly well, will keep you posted.

[tomkail]

Oh my! This is absolutely fantastic, thank you so much for writing this up! Would you mind if I shared it on our discord? Would love to see any updates, like the string comparison idea you mentioned.

As for tags - we added the ability to add tags to choices last year - that might be it?